admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(document-processors): escalate glyph-corrupt PDFs to the structured tier

Browse Source 
The fast (pypdfium2) extractor can leak raw glyph codes on subset fonts with a
broken /ToUnicode CMap. The result scores high on the existing text-quality
heuristic -- a uniform glyph/Caesar offset preserves whitespace and token
lengths -- yet is unsearchable. The structured (pymupdf) tier extracts the same
pages correctly.

Add a language-agnostic C0-control-character-ratio signal to the tier-0
classifier that detects this corruption and routes the document to a new
`structured` recommended_tier. Wire the fast->structured hop on the inline path
and generalise it so a low-quality-but-non-empty layer also tries structured
before OCR -- the inline and external ingest modes now follow the full
fast->structured->ocr ladder identically. A scanned / no-text-layer document
(total_chars == 0) still shortcuts straight to OCR, since a text extractor
cannot recover a pure raster.

New per-tenant tunable DOCUMENT_GLYPH_CORRUPTION_RATIO (default 0.02); escalation
metrics gain a `corrupt_glyphs` reason label.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-16 20:06:35 +02:00
co-authored by Claude Opus 4.8
parent 60df141442
commit cf7209cd85
8 changed files with 412 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/observability/metrics.py
+2 -2
View File
@@ -272,7 +272,7 @@ document_bytes_processed_total = Counter(
document_escalation_total = Counter(
"astrolabe_document_escalation_total",
"Total document parse escalations between tiers",
# reason: low_confidence | empty_text | unsupported | error | forced
# reason: low_confidence | empty_text | corrupt_glyphs | unsupported | error | forced
["from_tier", "to_tier", "reason"],
)
@@ -754,7 +754,7 @@ def record_document_escalation(from_tier: str, to_tier: str, reason: str) -> Non
Args:
from_tier: Tier that could not satisfactorily parse the document
to_tier: Tier the document was escalated to
reason: low_confidence | empty_text | unsupported | error | forced
reason: low_confidence | empty_text | corrupt_glyphs | unsupported | error | forced
"""
document_escalation_total.labels(
from_tier=from_tier, to_tier=to_tier, reason=reason