feat: quality + scan OCR escalation trigger (junk-text-layer scans)
The hot-path classifier escalated to OCR purely on character count, so a scanned/handwritten PDF with a low-quality embedded text layer (>16 chars/page but garbled) routed `fast` and indexed the junk -- e.g. Student 147.pdf's "Little Acoms Primary"/"0110912020", which pollutes the vector and demotes the doc in search (Deck #207). - classifier: recalibrate `_text_quality` with a long-token-fraction term that detects word-merging (dropped inter-word spaces) -- the dominant junk-layer failure the old whitespace/overlong(>20) terms missed. Measured: the Student 147 scan ~0.42 (60% pages junk) vs >=0.94 for clean digital docs. - classify_from_text now routes on quality + scan: a page is OCR-worthy if near-empty OR low text-quality OR (when OCR + scan detection are enabled) it's mostly a raster image. New `image_coverage_per_page` re-opens the PDF for the scan signal, so that cost is paid only by OCR-opted-in tenants. Thresholds are passed in from per-tenant settings (keyword-only). - config: 4 per-tenant settings -- DOCUMENT_OCR_MIN_TEXT_QUALITY (0.5), DOCUMENT_OCR_PAGE_FRACTION (0.5), DOCUMENT_OCR_MIN_PAGE_CHARS (16), DOCUMENT_OCR_DETECT_SCANNED (true) -- with range validators. - metrics: new astrolabe_document_ocr_page_fraction histogram (the value the page-fraction threshold acts on) alongside document_text_quality, so operators can tune the OCR escalation per tenant (quality vs cost). Escalation gate, OCR backends, and off-by-default behavior unchanged (#858). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d421bf6953
commit
b1f347b8fc
@@ -381,6 +381,22 @@ class TestDynaconfValidators:
|
||||
with pytest.raises(ValidationError, match="LOG_FORMAT"):
|
||||
_reload_config()
|
||||
|
||||
@patch.dict(os.environ, {"DOCUMENT_OCR_MIN_TEXT_QUALITY": "1.5"}, clear=True)
|
||||
def test_ocr_min_text_quality_out_of_range(self):
|
||||
"""DOCUMENT_OCR_MIN_TEXT_QUALITY must be in [0, 1]."""
|
||||
from dynaconf import ValidationError
|
||||
|
||||
with pytest.raises(ValidationError, match="DOCUMENT_OCR_MIN_TEXT_QUALITY"):
|
||||
_reload_config()
|
||||
|
||||
@patch.dict(os.environ, {"DOCUMENT_OCR_PAGE_FRACTION": "2"}, clear=True)
|
||||
def test_ocr_page_fraction_out_of_range(self):
|
||||
"""DOCUMENT_OCR_PAGE_FRACTION must be in [0, 1]."""
|
||||
from dynaconf import ValidationError
|
||||
|
||||
with pytest.raises(ValidationError, match="DOCUMENT_OCR_PAGE_FRACTION"):
|
||||
_reload_config()
|
||||
|
||||
@patch.dict(os.environ, {"LOG_LEVEL": "VERBOSE"}, clear=True)
|
||||
def test_invalid_log_level(self):
|
||||
"""Test invalid LOG_LEVEL raises ValidationError."""
|
||||
|
||||
Reference in New Issue
Block a user