docs(document-processors): round-2 review nits + classify_pdf glyph test
Address round-2 review on PR #914: - Add corrupt_glyphs to the document_classifier_flag_total label comment (it is a live flag value emitted by record_document_classification). - Mirror the full_text-vs-sampled control-ratio NOTE into classify_pdf so the diagnostic path's under-detection trade-off is documented in place. - Add test_classify_pdf_glyph_corrupt_routes_structured for routing symmetry on the standalone classify_pdf path. (SonarCloud quality gate is green — the prior S1244 finding was fixed last round.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d5286e39d6
commit
425eb839bf
@@ -272,6 +272,10 @@ def classify_pdf(content: bytes) -> DocClassification:
|
||||
ocr_frac = (sum(p.needs_ocr for p in pages) / sampled) if sampled else 0.0
|
||||
# Char-weighted doc-level control-char ratio (p.control_ratio * char_count is
|
||||
# the per-page bad-char count). The glyph-leak signal -- see _control_char_ratio.
|
||||
# NOTE: this is over the <=MAX_SAMPLED_PAGES sample, so unlike classify_from_text
|
||||
# (which scans the whole full_text) this diagnostic path can under-detect
|
||||
# corruption concentrated outside the sampled pages. Acceptable here: the hot
|
||||
# path is classify_from_text; this standalone pass is for diagnostics.
|
||||
control_ratio = (
|
||||
sum(p.control_ratio * p.char_count for p in pages) / total_chars
|
||||
if total_chars
|
||||
|
||||
@@ -334,7 +334,7 @@ document_classifier_flag_total = Counter(
|
||||
# so flag{image_heavy} is expected to exceed classified{recommended_tier=ocr}.
|
||||
"astrolabe_document_classifier_flag_total",
|
||||
"Tier-0 classifier flags raised on documents",
|
||||
["flag"], # image_heavy | scanned | bad_text_layer
|
||||
["flag"], # image_heavy | scanned | bad_text_layer | corrupt_glyphs
|
||||
)
|
||||
|
||||
document_text_quality = Histogram(
|
||||
|
||||
Reference in New Issue
Block a user