fix(document-processors): make glyph-corruption ratio of 0 disable the signal
Address round-4 review on PR #914: - glyph_corruption_ratio <= 0 now disables the signal (previously `control_ratio > 0` fired on any single C0 control byte), matching the "0 disables" convention used elsewhere (document_max_pdf_size_mb) and the config comment. Add a zero-disables test. - Correct the document_escalation_suppressed_total comment: corrupt_glyphs CAN appear there in the narrow case where structured is unregistered and OCR is registered-but-disabled (evaluate_escalation follows minimum="structured" past the missing rung to a gated-off OCR). Add a test for that suppressed decision. - Add a test for the double-corruption edge: a structured re-extract that is also glyph-corrupt escalates structured->ocr with reason corrupt_glyphs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
33aadbcf80
commit
4af7c7104b
@@ -203,9 +203,15 @@ def _route_from_signals(
|
||||
Flags are diagnostic and independent of the verdict (e.g. ``image_heavy``
|
||||
fires on ANY image-heavy page; the OCR route needs a page FRACTION).
|
||||
"""
|
||||
# total_chars > 0 also guarantees this never overlaps the scanned branch
|
||||
# (total_chars == 0), so a doc is never both glyph-corrupt and "scanned".
|
||||
glyph_corrupt = total_chars > 0 and control_ratio > glyph_corruption_ratio
|
||||
# glyph_corruption_ratio <= 0 disables the signal (a ratio of 0 would otherwise
|
||||
# fire on any single C0 control byte). total_chars > 0 also guarantees this
|
||||
# never overlaps the scanned branch (total_chars == 0), so a doc is never both
|
||||
# glyph-corrupt and "scanned".
|
||||
glyph_corrupt = (
|
||||
glyph_corruption_ratio > 0
|
||||
and total_chars > 0
|
||||
and control_ratio > glyph_corruption_ratio
|
||||
)
|
||||
|
||||
flags: set[str] = set()
|
||||
if ocr_frac >= page_fraction and total_chars == 0:
|
||||
|
||||
@@ -287,9 +287,10 @@ document_escalation_total = Counter(
|
||||
document_escalation_suppressed_total = Counter(
|
||||
"astrolabe_document_escalation_suppressed_total",
|
||||
"Would-be tier escalations suppressed because the target tier is disabled",
|
||||
# reason: low_confidence | empty_text. (corrupt_glyphs never appears here: it
|
||||
# targets the structured tier, which has no enabled-gate -- if registered it
|
||||
# runs, else there is nothing to suppress -- so it only ever hops or returns.)
|
||||
# reason: low_confidence | empty_text | corrupt_glyphs. (corrupt_glyphs lands
|
||||
# here only in the narrow case where the structured tier is unregistered AND
|
||||
# OCR is registered-but-disabled: evaluate_escalation follows minimum="structured"
|
||||
# past the missing rung to OCR, which is gated off -> suppressed{to_tier="ocr"}.)
|
||||
["from_tier", "to_tier", "reason"],
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user