fix(review): align classify_pdf routing with the hot path + scan-tail test
Address PR #863 round 2: - classify_pdf now flags a page needs_ocr on the SAME three signals as classify_from_text (image scan OR low text-quality OR near-empty), not image coverage alone. Previously a word-merged digital doc with no images routed "fast" via classify_pdf but "ocr" via the pipeline -- so an operator reproducing routing offline got a different answer. They now match. - Add a test that when image_coverage is shorter than the page boundaries (the MAX_SAMPLED_PAGES cap on large scans), the leading page uses the scan signal and later pages fall back to text-quality. Left as-is: overlong_score (>20) partially overlaps merge_score (>12) -- the double-penalty on very-long tokens is intentional, not a bug (per review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
fbc9a3a675
commit
0287bd9175
@@ -152,13 +152,15 @@ def classify_pdf(content: bytes) -> DocClassification:
|
||||
text = page.get_text("text")
|
||||
quality = _text_quality(text)
|
||||
coverage = _page_image_coverage(page)
|
||||
# A page that is mostly a raster image is a scan/photo: its content
|
||||
# (handwriting, stamps, figure text) is not fully in any text layer,
|
||||
# so OCR is needed to capture it -- regardless of whether a partial
|
||||
# text layer is present. Text quality/char-count are kept as
|
||||
# diagnostic signals (flags + tuning metrics), not the trigger,
|
||||
# because OCR only helps when there is an image to read.
|
||||
needs_ocr = coverage >= IMAGE_COVERAGE_SCANNED
|
||||
# OCR-worthy on the same three signals as classify_from_text (kept in
|
||||
# sync so an operator reproducing routing offline gets the pipeline's
|
||||
# answer): a mostly-raster scan, a junk/low-quality text layer (the
|
||||
# word-merging case), or an effectively empty text layer.
|
||||
needs_ocr = (
|
||||
coverage >= IMAGE_COVERAGE_SCANNED
|
||||
or quality < MIN_TEXT_QUALITY
|
||||
or len(text.strip()) < MIN_PAGE_CHARS
|
||||
)
|
||||
pages.append(
|
||||
PageSignals(n, len(text), round(coverage, 3), quality, needs_ocr)
|
||||
)
|
||||
|
||||
@@ -268,3 +268,19 @@ def test_image_coverage_per_page():
|
||||
assert len(scan) == 2 and all(c >= 0.8 for c in scan)
|
||||
digital = clf.image_coverage_per_page(_digital_pdf(pages=2))
|
||||
assert len(digital) == 2 and all(c < 0.1 for c in digital)
|
||||
|
||||
|
||||
def test_scan_coverage_shorter_than_pages_falls_back_to_text():
|
||||
# image_coverage shorter than the boundaries (the MAX_SAMPLED_PAGES cap):
|
||||
# page 0 is flagged scanned; later pages fall back to the text-quality signal.
|
||||
n = len(_CLEAN)
|
||||
full = _CLEAN * 3
|
||||
bounds = [
|
||||
{"page": 1, "start_offset": 0, "end_offset": n},
|
||||
{"page": 2, "start_offset": n, "end_offset": 2 * n},
|
||||
{"page": 3, "start_offset": 2 * n, "end_offset": 3 * n},
|
||||
]
|
||||
c = clf.classify_from_text(full, bounds, image_coverage=[1.0])
|
||||
assert c.pages[0].needs_ocr is True # scanned (coverage)
|
||||
assert c.pages[1].needs_ocr is False # clean text, no coverage entry
|
||||
assert c.recommended_tier == "fast" # only 1/3 pages bad
|
||||
|
||||
Reference in New Issue
Block a user