fix(review): align classify_pdf routing with the hot path + scan-tail test

Address PR #863 round 2:

- classify_pdf now flags a page needs_ocr on the SAME three signals as
  classify_from_text (image scan OR low text-quality OR near-empty), not image
  coverage alone. Previously a word-merged digital doc with no images routed
  "fast" via classify_pdf but "ocr" via the pipeline -- so an operator
  reproducing routing offline got a different answer. They now match.
- Add a test that when image_coverage is shorter than the page boundaries (the
  MAX_SAMPLED_PAGES cap on large scans), the leading page uses the scan signal
  and later pages fall back to text-quality.

Left as-is: overlong_score (>20) partially overlaps merge_score (>12) -- the
double-penalty on very-long tokens is intentional, not a bug (per review).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 05:01:09 +02:00
co-authored by Claude Opus 4.8
parent fbc9a3a675
commit 0287bd9175
2 changed files with 25 additions and 7 deletions
+16
View File
@@ -268,3 +268,19 @@ def test_image_coverage_per_page():
assert len(scan) == 2 and all(c >= 0.8 for c in scan)
digital = clf.image_coverage_per_page(_digital_pdf(pages=2))
assert len(digital) == 2 and all(c < 0.1 for c in digital)
def test_scan_coverage_shorter_than_pages_falls_back_to_text():
# image_coverage shorter than the boundaries (the MAX_SAMPLED_PAGES cap):
# page 0 is flagged scanned; later pages fall back to the text-quality signal.
n = len(_CLEAN)
full = _CLEAN * 3
bounds = [
{"page": 1, "start_offset": 0, "end_offset": n},
{"page": 2, "start_offset": n, "end_offset": 2 * n},
{"page": 3, "start_offset": 2 * n, "end_offset": 3 * n},
]
c = clf.classify_from_text(full, bounds, image_coverage=[1.0])
assert c.pages[0].needs_ocr is True # scanned (coverage)
assert c.pages[1].needs_ocr is False # clean text, no coverage entry
assert c.recommended_tier == "fast" # only 1/3 pages bad