Chris CoutinhoandClaude Opus 4.8
044c1da750
First step of the tiered document-processor effort (Deck #203): a cheap, local pre-pass that recommends which extraction tier a PDF should start in, emitting metrics WITHOUT changing routing yet -- so we gather per-tenant doc-mix data before turning escalation on. document_processors/classifier.py: classify_pdf(content) -> DocClassification. Page-sampled (bounded on large docs), <~1s. Cheap signals only -- text-layer chars, a text-quality score (catches the "Student 147" failure where a text layer exists but is mashed/space-less junk), and image coverage. A page that is mostly a raster image routes to OCR: its content (handwriting, stamps) isn't in any text layer. Deliberately no get_drawings/graphics-density signal -- it's slow on the exact pages it'd flag, the hotfix's graphics_limit already makes the parse safe, and the (future) tier-1 quality gate catches lost tables. Validated on the sample corpus: born-digital 2-col arxiv and a digital student record -> fast (tier 1); a scanned+handwritten form -> ocr (tier 3). Wiring (vector/processor.py): _shadow_classify runs the classifier on PDFs in a worker thread, best-effort (never blocks/fails indexing), gated by the new DOCUMENT_CLASSIFY_ENABLED setting. Metrics: astrolabe_document_classified_total {recommended_tier}, astrolabe_document_classifier_flag_total{flag}, astrolabe_document_text_quality histogram. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
87 lines
2.6 KiB
Python
87 lines
2.6 KiB
Python
"""Unit tests for the tier-0 document classifier.
|
|
|
|
Pins the routing decisions and the text-quality heuristic that drive which
|
|
extraction tier a PDF starts in:
|
|
* a clean born-digital PDF (text, no full-page images) -> ``fast`` (tier 1);
|
|
* a full-page-image scan -> ``ocr`` (tier 3), since handwriting/stamps aren't
|
|
in any text layer;
|
|
* the text-quality score distinguishes clean prose from mashed/space-less junk.
|
|
"""
|
|
|
|
import pymupdf
|
|
import pytest
|
|
|
|
from nextcloud_mcp_server.document_processors import classifier as clf
|
|
|
|
pytestmark = pytest.mark.unit
|
|
|
|
|
|
def _digital_pdf(
|
|
pages: int = 3, body: str = "Hello world this is clean text. "
|
|
) -> bytes:
|
|
doc = pymupdf.open()
|
|
for _ in range(pages):
|
|
page = doc.new_page(width=595, height=842)
|
|
page.insert_text((50, 60), body * 8)
|
|
data: bytes = doc.tobytes()
|
|
doc.close()
|
|
return data
|
|
|
|
|
|
def _full_page_image_pdf(pages: int = 2) -> bytes:
|
|
# A page whose entire area is a raster image -> looks scanned.
|
|
doc = pymupdf.open()
|
|
pix = pymupdf.Pixmap(pymupdf.csRGB, pymupdf.IRect(0, 0, 600, 850))
|
|
pix.clear_with(255)
|
|
img = pix.tobytes("png")
|
|
for _ in range(pages):
|
|
page = doc.new_page(width=595, height=842)
|
|
page.insert_image(page.rect, stream=img)
|
|
data: bytes = doc.tobytes()
|
|
doc.close()
|
|
return data
|
|
|
|
|
|
# --- text-quality heuristic --------------------------------------------------
|
|
|
|
|
|
def test_text_quality_clean_prose_scores_high():
|
|
assert clf._text_quality("the quick brown fox jumps over the lazy dog") > 0.8
|
|
|
|
|
|
def test_text_quality_mashed_tokens_scores_low():
|
|
# space-less / mashed layer (the "Student 147" failure mode)
|
|
mashed = "01322234567mobileoutstandingresilienceacademicachievementhurdles"
|
|
assert clf._text_quality(mashed) < clf.MIN_TEXT_QUALITY
|
|
|
|
|
|
def test_text_quality_empty_is_zero():
|
|
assert clf._text_quality("") == 0.0
|
|
|
|
|
|
# --- routing -----------------------------------------------------------------
|
|
|
|
|
|
def test_digital_pdf_routes_fast():
|
|
c = clf.classify_pdf(_digital_pdf())
|
|
assert c.recommended_tier == "fast"
|
|
assert c.ocr_page_fraction == 0.0
|
|
assert "image_heavy" not in c.flags
|
|
assert c.mean_text_quality > 0.8
|
|
|
|
|
|
def test_full_page_image_routes_ocr():
|
|
c = clf.classify_pdf(_full_page_image_pdf())
|
|
assert c.recommended_tier == "ocr"
|
|
assert c.ocr_page_fraction == 1.0
|
|
assert "image_heavy" in c.flags
|
|
|
|
|
|
# --- sampling bounds large docs ----------------------------------------------
|
|
|
|
|
|
def test_large_doc_is_sampled():
|
|
c = clf.classify_pdf(_digital_pdf(pages=120))
|
|
assert c.page_count == 120
|
|
assert c.sampled_pages <= clf.MAX_SAMPLED_PAGES
|