fix(review): align classify_pdf routing with the hot path + scan-tail test

Address PR #863 round 2:

- classify_pdf now flags a page needs_ocr on the SAME three signals as
  classify_from_text (image scan OR low text-quality OR near-empty), not image
  coverage alone. Previously a word-merged digital doc with no images routed
  "fast" via classify_pdf but "ocr" via the pipeline -- so an operator
  reproducing routing offline got a different answer. They now match.
- Add a test that when image_coverage is shorter than the page boundaries (the
  MAX_SAMPLED_PAGES cap on large scans), the leading page uses the scan signal
  and later pages fall back to text-quality.

Left as-is: overlong_score (>20) partially overlaps merge_score (>12) -- the
double-penalty on very-long tokens is intentional, not a bug (per review).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 05:01:09 +02:00
co-authored by Claude Opus 4.8
parent fbc9a3a675
commit 0287bd9175
2 changed files with 25 additions and 7 deletions
@@ -152,13 +152,15 @@ def classify_pdf(content: bytes) -> DocClassification:
text = page.get_text("text")
quality = _text_quality(text)
coverage = _page_image_coverage(page)
# A page that is mostly a raster image is a scan/photo: its content
# (handwriting, stamps, figure text) is not fully in any text layer,
# so OCR is needed to capture it -- regardless of whether a partial
# text layer is present. Text quality/char-count are kept as
# diagnostic signals (flags + tuning metrics), not the trigger,
# because OCR only helps when there is an image to read.
needs_ocr = coverage >= IMAGE_COVERAGE_SCANNED
# OCR-worthy on the same three signals as classify_from_text (kept in
# sync so an operator reproducing routing offline gets the pipeline's
# answer): a mostly-raster scan, a junk/low-quality text layer (the
# word-merging case), or an effectively empty text layer.
needs_ocr = (
coverage >= IMAGE_COVERAGE_SCANNED
or quality < MIN_TEXT_QUALITY
or len(text.strip()) < MIN_PAGE_CHARS
)
pages.append(
PageSignals(n, len(text), round(coverage, 3), quality, needs_ocr)
)