fix(document-processors): inline/external parity when structured tier is absent

Address round-3 review on PR #914:
- When a glyph-corrupt doc's structured rung is NOT registered, the inline path
  now falls through to OCR (with reason corrupt_glyphs), mirroring the external
  next_available_tier instead of silently keeping the fast result. A structured
  parse FAILURE remains terminal (tracked via structured_failed), matching the
  external path which does not escalate a failure. Added a debug log for the
  unregistered case and "(OCR not attempted)" to the failure warning.
- Tests: inline + external glyph-corrupt fallthrough to OCR when structured is
  unregistered; glyph-corrupt + junk-quality both-flags precedence (structured
  wins over the bad_text_layer/ocr route).
- Note the total_chars>0 mutual-exclusion with the scanned branch in
  _route_from_signals.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-16 20:34:07 +02:00
co-authored by Claude Opus 4.8
parent 24c22f19d6
commit 33aadbcf80
4 changed files with 91 additions and 12 deletions
+15
View File
@@ -395,6 +395,21 @@ def test_empty_doc_routes_ocr_not_structured():
assert "corrupt_glyphs" not in c.flags
def test_glyph_corrupt_takes_precedence_over_junk_text_layer():
# A layer that is BOTH glyph-corrupt (high control ratio) AND junk-quality
# (mashed, no whitespace -> low text_quality): both flags fire, but
# glyph-corrupt wins the route (structured, not ocr) -- the structured
# re-extract is the cheaper correct fix, and re-classification catches any
# residual junk afterwards.
text = "WKHTXLFNEURZQIRAMXPSV\x0f\x10\x11\x0f\x10" * 3
c = clf.classify_from_text(
text, [{"page": 1, "start_offset": 0, "end_offset": len(text)}]
)
assert c.recommended_tier == "structured"
assert "corrupt_glyphs" in c.flags
assert "bad_text_layer" in c.flags
def test_classify_pdf_glyph_corrupt_routes_structured():
# Symmetry with the classify_from_text routing on the standalone/diagnostic
# classify_pdf path (which re-opens the PDF and samples pages).