fix(document-processors): inline/external parity when structured tier is absent
Address round-3 review on PR #914: - When a glyph-corrupt doc's structured rung is NOT registered, the inline path now falls through to OCR (with reason corrupt_glyphs), mirroring the external next_available_tier instead of silently keeping the fast result. A structured parse FAILURE remains terminal (tracked via structured_failed), matching the external path which does not escalate a failure. Added a debug log for the unregistered case and "(OCR not attempted)" to the failure warning. - Tests: inline + external glyph-corrupt fallthrough to OCR when structured is unregistered; glyph-corrupt + junk-quality both-flags precedence (structured wins over the bad_text_layer/ocr route). - Note the total_chars>0 mutual-exclusion with the scanned branch in _route_from_signals. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
24c22f19d6
commit
33aadbcf80
@@ -395,6 +395,21 @@ def test_empty_doc_routes_ocr_not_structured():
|
||||
assert "corrupt_glyphs" not in c.flags
|
||||
|
||||
|
||||
def test_glyph_corrupt_takes_precedence_over_junk_text_layer():
|
||||
# A layer that is BOTH glyph-corrupt (high control ratio) AND junk-quality
|
||||
# (mashed, no whitespace -> low text_quality): both flags fire, but
|
||||
# glyph-corrupt wins the route (structured, not ocr) -- the structured
|
||||
# re-extract is the cheaper correct fix, and re-classification catches any
|
||||
# residual junk afterwards.
|
||||
text = "WKHTXLFNEURZQIRAMXPSV\x0f\x10\x11\x0f\x10" * 3
|
||||
c = clf.classify_from_text(
|
||||
text, [{"page": 1, "start_offset": 0, "end_offset": len(text)}]
|
||||
)
|
||||
assert c.recommended_tier == "structured"
|
||||
assert "corrupt_glyphs" in c.flags
|
||||
assert "bad_text_layer" in c.flags
|
||||
|
||||
|
||||
def test_classify_pdf_glyph_corrupt_routes_structured():
|
||||
# Symmetry with the classify_from_text routing on the standalone/diagnostic
|
||||
# classify_pdf path (which re-opens the PDF and samples pages).
|
||||
|
||||
Reference in New Issue
Block a user