feat: quality + scan OCR escalation trigger (junk-text-layer scans)
The hot-path classifier escalated to OCR purely on character count, so a scanned/handwritten PDF with a low-quality embedded text layer (>16 chars/page but garbled) routed `fast` and indexed the junk -- e.g. Student 147.pdf's "Little Acoms Primary"/"0110912020", which pollutes the vector and demotes the doc in search (Deck #207). - classifier: recalibrate `_text_quality` with a long-token-fraction term that detects word-merging (dropped inter-word spaces) -- the dominant junk-layer failure the old whitespace/overlong(>20) terms missed. Measured: the Student 147 scan ~0.42 (60% pages junk) vs >=0.94 for clean digital docs. - classify_from_text now routes on quality + scan: a page is OCR-worthy if near-empty OR low text-quality OR (when OCR + scan detection are enabled) it's mostly a raster image. New `image_coverage_per_page` re-opens the PDF for the scan signal, so that cost is paid only by OCR-opted-in tenants. Thresholds are passed in from per-tenant settings (keyword-only). - config: 4 per-tenant settings -- DOCUMENT_OCR_MIN_TEXT_QUALITY (0.5), DOCUMENT_OCR_PAGE_FRACTION (0.5), DOCUMENT_OCR_MIN_PAGE_CHARS (16), DOCUMENT_OCR_DETECT_SCANNED (true) -- with range validators. - metrics: new astrolabe_document_ocr_page_fraction histogram (the value the page-fraction threshold acts on) alongside document_text_quality, so operators can tune the OCR escalation per tenant (quality vs cost). Escalation gate, OCR backends, and off-by-default behavior unchanged (#858). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
d421bf6953
commit
b1f347b8fc
@@ -299,6 +299,18 @@ document_text_quality = Histogram(
|
||||
buckets=(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0),
|
||||
)
|
||||
|
||||
# Per-document fraction of OCR-worthy pages (near-empty / junk-quality / scanned).
|
||||
# This is the value the DOCUMENT_OCR_PAGE_FRACTION threshold acts on, so its
|
||||
# distribution per tenant is the lever for tuning OCR escalation (quality vs
|
||||
# cost): how many docs sit just below/above the cutoff. Pair with
|
||||
# document_text_quality (where to set the per-page quality floor) and
|
||||
# document_escalation_total (realized OCR volume).
|
||||
document_ocr_page_fraction = Histogram(
|
||||
"astrolabe_document_ocr_page_fraction",
|
||||
"Tier-0 fraction of OCR-worthy pages per document (0=all-clean, 1=all-bad)",
|
||||
buckets=(0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0),
|
||||
)
|
||||
|
||||
# --- Embedding stages ---------------------------------------------------------
|
||||
|
||||
embedding_duration_seconds = Histogram(
|
||||
@@ -665,17 +677,23 @@ def record_document_parse_failed(reason: str) -> None:
|
||||
|
||||
|
||||
def record_document_classification(
|
||||
recommended_tier: str, flags: set[str], mean_text_quality: float
|
||||
recommended_tier: str,
|
||||
flags: set[str],
|
||||
mean_text_quality: float,
|
||||
ocr_page_fraction: float = 0.0,
|
||||
) -> None:
|
||||
"""Record a tier-0 classification result (shadow mode -- observability only).
|
||||
"""Record a tier-0 classification result.
|
||||
|
||||
Primitive args (not the DocClassification object) keep the observability
|
||||
layer free of a dependency on document_processors.
|
||||
layer free of a dependency on document_processors. ``mean_text_quality`` and
|
||||
``ocr_page_fraction`` feed the two histograms operators use to tune the OCR
|
||||
escalation thresholds per tenant (quality vs cost).
|
||||
"""
|
||||
document_classified_total.labels(recommended_tier=recommended_tier).inc()
|
||||
for flag in flags:
|
||||
document_classifier_flag_total.labels(flag=flag).inc()
|
||||
document_text_quality.observe(mean_text_quality)
|
||||
document_ocr_page_fraction.observe(ocr_page_fraction)
|
||||
|
||||
|
||||
def record_embedding(
|
||||
|
||||
Reference in New Issue
Block a user