fix(document-processors): make glyph-corruption ratio of 0 disable the signal
Address round-4 review on PR #914: - glyph_corruption_ratio <= 0 now disables the signal (previously `control_ratio > 0` fired on any single C0 control byte), matching the "0 disables" convention used elsewhere (document_max_pdf_size_mb) and the config comment. Add a zero-disables test. - Correct the document_escalation_suppressed_total comment: corrupt_glyphs CAN appear there in the narrow case where structured is unregistered and OCR is registered-but-disabled (evaluate_escalation follows minimum="structured" past the missing rung to a gated-off OCR). Add a test for that suppressed decision. - Add a test for the double-corruption edge: a structured re-extract that is also glyph-corrupt escalates structured->ocr with reason corrupt_glyphs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
33aadbcf80
commit
4af7c7104b
@@ -387,6 +387,15 @@ def test_glyph_corruption_ratio_override_disables_trigger():
|
||||
assert "corrupt_glyphs" not in c.flags
|
||||
|
||||
|
||||
def test_glyph_corruption_ratio_zero_disables_trigger():
|
||||
full = _GLYPH_CORRUPT
|
||||
bounds = [{"page": 1, "start_offset": 0, "end_offset": len(full)}]
|
||||
# 0 disables the signal (rather than firing on any single control byte).
|
||||
c = clf.classify_from_text(full, bounds, glyph_corruption_ratio=0.0)
|
||||
assert c.recommended_tier == "fast"
|
||||
assert "corrupt_glyphs" not in c.flags
|
||||
|
||||
|
||||
def test_empty_doc_routes_ocr_not_structured():
|
||||
# Precedence: a scanned/empty doc (no text layer) has no control chars to leak,
|
||||
# so it must stay an OCR case, never structured.
|
||||
|
||||
Reference in New Issue
Block a user