feat: quality + scan OCR escalation trigger (junk-text-layer scans)
The hot-path classifier escalated to OCR purely on character count, so a scanned/handwritten PDF with a low-quality embedded text layer (>16 chars/page but garbled) routed `fast` and indexed the junk -- e.g. Student 147.pdf's "Little Acoms Primary"/"0110912020", which pollutes the vector and demotes the doc in search (Deck #207). - classifier: recalibrate `_text_quality` with a long-token-fraction term that detects word-merging (dropped inter-word spaces) -- the dominant junk-layer failure the old whitespace/overlong(>20) terms missed. Measured: the Student 147 scan ~0.42 (60% pages junk) vs >=0.94 for clean digital docs. - classify_from_text now routes on quality + scan: a page is OCR-worthy if near-empty OR low text-quality OR (when OCR + scan detection are enabled) it's mostly a raster image. New `image_coverage_per_page` re-opens the PDF for the scan signal, so that cost is paid only by OCR-opted-in tenants. Thresholds are passed in from per-tenant settings (keyword-only). - config: 4 per-tenant settings -- DOCUMENT_OCR_MIN_TEXT_QUALITY (0.5), DOCUMENT_OCR_PAGE_FRACTION (0.5), DOCUMENT_OCR_MIN_PAGE_CHARS (16), DOCUMENT_OCR_DETECT_SCANNED (true) -- with range validators. - metrics: new astrolabe_document_ocr_page_fraction histogram (the value the page-fraction threshold acts on) alongside document_text_quality, so operators can tune the OCR escalation per tenant (quality vs cost). Escalation gate, OCR backends, and off-by-default behavior unchanged (#858). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
d421bf6953
commit
b1f347b8fc
@@ -149,6 +149,15 @@ _DEFAULTS: dict[str, Any] = {
|
||||
# Provider-namespaced OCR model id (gateway routes on the prefix; the direct
|
||||
# mistral backend strips it).
|
||||
"document_ocr_model": "mistral/mistral-ocr-latest",
|
||||
# OCR escalation triggers (tier-0). A page is OCR-worthy when its text is
|
||||
# near-empty (< min_page_chars) OR low-quality (< min_text_quality) OR (when
|
||||
# detect_scanned) mostly a raster image; a doc escalates when the OCR-worthy
|
||||
# page fraction reaches page_fraction. Calibrate min_text_quality from the
|
||||
# astrolabe_document_text_quality histogram per tenant.
|
||||
"document_ocr_min_text_quality": 0.5,
|
||||
"document_ocr_page_fraction": 0.5,
|
||||
"document_ocr_min_page_chars": 16,
|
||||
"document_ocr_detect_scanned": True,
|
||||
# Observability
|
||||
"metrics_enabled": True,
|
||||
"metrics_port": 9090,
|
||||
@@ -297,6 +306,10 @@ _dynaconf = Dynaconf(
|
||||
# >=1: pymupdf4llm treats graphics_limit=0 as "no cap", which would
|
||||
# re-expose the OOM this guards against.
|
||||
Validator("DOCUMENT_PDF_GRAPHICS_LIMIT", gte=1),
|
||||
# OCR escalation thresholds: quality + page-fraction are [0, 1].
|
||||
Validator("DOCUMENT_OCR_MIN_TEXT_QUALITY", gte=0, lte=1),
|
||||
Validator("DOCUMENT_OCR_PAGE_FRACTION", gte=0, lte=1),
|
||||
Validator("DOCUMENT_OCR_MIN_PAGE_CHARS", gte=0),
|
||||
# Non-negative
|
||||
Validator("DOCUMENT_CHUNK_OVERLAP", gte=0),
|
||||
# Non-empty strings
|
||||
@@ -757,6 +770,14 @@ class Settings:
|
||||
# gateway routes on the "<provider>/" prefix; the direct mistral backend
|
||||
# strips it.
|
||||
document_ocr_model: str = "mistral/mistral-ocr-latest"
|
||||
# OCR escalation triggers (tier-0), per-tenant tunable. A page is OCR-worthy
|
||||
# if near-empty (< min_page_chars) OR low text-quality (< min_text_quality)
|
||||
# OR (when detect_scanned, image-analysis only runs when OCR is enabled)
|
||||
# mostly a raster image; the doc escalates at >= page_fraction such pages.
|
||||
document_ocr_min_text_quality: float = 0.5
|
||||
document_ocr_page_fraction: float = 0.5
|
||||
document_ocr_min_page_chars: int = 16
|
||||
document_ocr_detect_scanned: bool = True
|
||||
|
||||
# Observability settings
|
||||
metrics_enabled: bool = True
|
||||
@@ -1376,6 +1397,10 @@ def get_settings() -> Settings:
|
||||
"document_ocr_enabled": "DOCUMENT_OCR_ENABLED",
|
||||
"document_ocr_provider": "DOCUMENT_OCR_PROVIDER",
|
||||
"document_ocr_model": "DOCUMENT_OCR_MODEL",
|
||||
"document_ocr_min_text_quality": "DOCUMENT_OCR_MIN_TEXT_QUALITY",
|
||||
"document_ocr_page_fraction": "DOCUMENT_OCR_PAGE_FRACTION",
|
||||
"document_ocr_min_page_chars": "DOCUMENT_OCR_MIN_PAGE_CHARS",
|
||||
"document_ocr_detect_scanned": "DOCUMENT_OCR_DETECT_SCANNED",
|
||||
# Observability settings
|
||||
"metrics_enabled": "METRICS_ENABLED",
|
||||
"metrics_port": "METRICS_PORT",
|
||||
|
||||
@@ -9,11 +9,14 @@ Decides which extraction tier a PDF should escalate to, from cheap signals:
|
||||
* no text layer -- the strongest OCR signal available from text alone.
|
||||
|
||||
Two entry points:
|
||||
* ``classify_from_text(text, page_boundaries)`` -- the HOT PATH. Derives the
|
||||
text-quality/no-text-layer signal from the text the registry's tier-1 step
|
||||
already extracted, so it adds ~no cost. No image analysis.
|
||||
* ``classify_from_text(text, page_boundaries, ...)`` -- the HOT PATH. Routes on
|
||||
text-quality + near-empty pages derived from the tier-1 extraction (~no
|
||||
cost). When OCR + scan-detection are enabled the registry also passes
|
||||
per-page ``image_coverage`` (from ``image_coverage_per_page``) so scans are
|
||||
caught too; that image pass is the only added cost and only OCR-opted-in
|
||||
tenants pay it. Thresholds come from per-tenant settings.
|
||||
* ``classify_pdf(content)`` -- a standalone/diagnostic pass that re-opens the
|
||||
PDF and adds image-coverage analysis. More expensive; used off the hot path.
|
||||
PDF and does image-coverage analysis inline. Off the hot path.
|
||||
|
||||
Recommended tier:
|
||||
* ``ocr`` -- scanned / no-usable-text-layer (route to tier 3, when enabled)
|
||||
@@ -83,6 +86,13 @@ def _text_quality(text: str) -> float:
|
||||
whitespace_ratio = sum(c.isspace() for c in text) / len(text)
|
||||
mean_token_len = sum(len(t) for t in tokens) / len(tokens)
|
||||
overlong_frac = sum(len(t) > 20 for t in tokens) / len(tokens)
|
||||
# Word-merging (dropped inter-word spaces) is the dominant junk-text-layer
|
||||
# failure mode on scanned forms -- the older whitespace/overlong(>20) terms
|
||||
# miss it, because the merges are 10-20 chars and a few dropped spaces still
|
||||
# leave whitespace above the 0.12 cap. Clean prose keeps <~3% of tokens above
|
||||
# 12 chars; merged/OCR-mangled layers push it past 10%. (Measured: the junk
|
||||
# Student-147 scan scores ~0.20 here vs >=0.9 for clean digital docs.)
|
||||
long_frac = sum(len(t) > 12 for t in tokens) / len(tokens)
|
||||
# Caps at 1.0 from 12% whitespace (conservative; clean prose runs 15-20%),
|
||||
# mean token ~4-6 chars, ~no overlong tokens.
|
||||
ws_score = min(whitespace_ratio / 0.12, 1.0)
|
||||
@@ -90,7 +100,8 @@ def _text_quality(text: str) -> float:
|
||||
1.0 if mean_token_len <= 10 else max(0.0, 1.0 - (mean_token_len - 10) / 15)
|
||||
)
|
||||
overlong_score = max(0.0, 1.0 - overlong_frac * 5)
|
||||
return round(ws_score * len_score * overlong_score, 3)
|
||||
merge_score = max(0.0, 1.0 - max(0.0, long_frac - 0.03) / 0.12)
|
||||
return round(ws_score * len_score * overlong_score * merge_score, 3)
|
||||
|
||||
|
||||
def _sample_indices(page_count: int) -> list[int]:
|
||||
@@ -180,27 +191,68 @@ def classify_pdf(content: bytes) -> DocClassification:
|
||||
)
|
||||
|
||||
|
||||
def classify_from_text(
|
||||
full_text: str, page_boundaries: list[dict[str, Any]]
|
||||
) -> DocClassification:
|
||||
"""Classify from text already extracted by tier-1 -- no PDF re-open.
|
||||
def image_coverage_per_page(content: bytes) -> list[float]:
|
||||
"""Raster-image coverage in ``[0, 1]`` for every page (document order).
|
||||
|
||||
The hot-path classifier: it derives the text-quality signal from the
|
||||
extraction the registry already ran, so it adds ~no cost (vs ``classify_pdf``,
|
||||
which re-opens the PDF and re-extracts). It does NOT do image analysis, so it
|
||||
cannot distinguish a scanned-with-text-layer page (that needs the image pass,
|
||||
which only matters once OCR routing is enabled). A page with effectively no
|
||||
text layer is the one OCR-worthy signal available from text alone.
|
||||
Lets the hot path flag scanned pages whose embedded text layer is junk but
|
||||
statistically clean-looking. Re-opens the PDF, so the registry calls it only
|
||||
when OCR + scan detection are enabled (the cost is borne by OCR-opted-in
|
||||
tenants). Returned list is aligned by index with the page boundaries.
|
||||
"""
|
||||
import pymupdf # noqa: PLC0415 -- keep the heavy import lazy
|
||||
|
||||
cov: list[float] = []
|
||||
with pymupdf.open("pdf", content) as doc:
|
||||
for n in range(doc.page_count):
|
||||
page = doc.load_page(n)
|
||||
page_area = abs(page.rect.width * page.rect.height) or 1.0
|
||||
img_area = 0.0
|
||||
for img in page.get_images(full=True):
|
||||
for rect in page.get_image_rects(img[0]):
|
||||
img_area += abs(rect.width * rect.height)
|
||||
cov.append(min(img_area / page_area, 1.0))
|
||||
return cov
|
||||
|
||||
|
||||
def classify_from_text(
|
||||
full_text: str,
|
||||
page_boundaries: list[dict[str, Any]],
|
||||
*,
|
||||
min_text_quality: float = MIN_TEXT_QUALITY,
|
||||
min_page_chars: int = MIN_PAGE_CHARS,
|
||||
page_fraction: float = OCR_PAGE_FRACTION,
|
||||
image_coverage: list[float] | None = None,
|
||||
) -> DocClassification:
|
||||
"""Classify from text already extracted by tier-1 -- no PDF re-open by default.
|
||||
|
||||
The hot-path classifier. A page is OCR-worthy when its text is near-empty
|
||||
(``< min_page_chars``), its text-quality is junk (``< min_text_quality`` --
|
||||
the word-merging signal), OR (when ``image_coverage`` is supplied, i.e. OCR +
|
||||
scan-detection are on) the page is mostly a raster image. The doc recommends
|
||||
``ocr`` once ``ocr_frac >= page_fraction``. Thresholds are passed in by the
|
||||
registry from per-tenant settings.
|
||||
|
||||
``page_boundaries`` are ``{page, start_offset, end_offset}`` indexing into
|
||||
``full_text`` (the tier-1/pdf_highlighter contract).
|
||||
``full_text``; ``image_coverage[i]`` (if given) aligns with the i-th boundary.
|
||||
"""
|
||||
pages: list[PageSignals] = []
|
||||
for b in page_boundaries:
|
||||
for idx, b in enumerate(page_boundaries):
|
||||
seg = full_text[b["start_offset"] : b["end_offset"]]
|
||||
needs_ocr = len(seg.strip()) < MIN_PAGE_CHARS
|
||||
quality = _text_quality(seg)
|
||||
# image_coverage is one entry per PDF page, aligned 1:1 with the
|
||||
# boundaries; the length guard is belt-and-suspenders against a mismatch.
|
||||
cov = (
|
||||
image_coverage[idx]
|
||||
if image_coverage is not None and idx < len(image_coverage)
|
||||
else 0.0
|
||||
)
|
||||
needs_ocr = (
|
||||
len(seg.strip()) < min_page_chars
|
||||
or quality < min_text_quality
|
||||
or cov >= IMAGE_COVERAGE_SCANNED
|
||||
)
|
||||
pages.append(
|
||||
PageSignals(b["page"], len(seg), 0.0, _text_quality(seg), needs_ocr)
|
||||
PageSignals(b["page"], len(seg), round(cov, 3), quality, needs_ocr)
|
||||
)
|
||||
|
||||
sampled = len(pages)
|
||||
@@ -213,18 +265,19 @@ def classify_from_text(
|
||||
# recorded classification metric accurate rather than a misleading "ocr").
|
||||
ocr_frac = (sum(p.needs_ocr for p in pages) / sampled) if sampled else 0.0
|
||||
|
||||
# Flags gated on ocr_frac >= OCR_PAGE_FRACTION (matching classify_pdf): a
|
||||
# doc that routes "fast" must not carry a junk-layer flag just because a few
|
||||
# isolated pages are bad -- otherwise the classification metric diverges
|
||||
# between this hot path and the standalone classify_pdf.
|
||||
# Flags gated on ocr_frac >= page_fraction (matching classify_pdf): a doc that
|
||||
# routes "fast" must not carry a junk-layer flag just because a few isolated
|
||||
# pages are bad -- otherwise the metric diverges from classify_pdf.
|
||||
flags: set[str] = set()
|
||||
if sampled and ocr_frac >= OCR_PAGE_FRACTION:
|
||||
if sampled and ocr_frac >= page_fraction:
|
||||
if total_chars == 0:
|
||||
flags.add("no_text_layer")
|
||||
elif mean_quality < MIN_TEXT_QUALITY:
|
||||
elif mean_quality < min_text_quality:
|
||||
flags.add("bad_text_layer")
|
||||
if any(p.image_coverage >= IMAGE_COVERAGE_SCANNED for p in pages):
|
||||
flags.add("image_heavy")
|
||||
|
||||
recommended = "ocr" if ocr_frac >= OCR_PAGE_FRACTION else "fast"
|
||||
recommended = "ocr" if ocr_frac >= page_fraction else "fast"
|
||||
|
||||
return DocClassification(
|
||||
page_count=len(page_boundaries),
|
||||
|
||||
@@ -14,7 +14,7 @@ from nextcloud_mcp_server.observability.metrics import (
|
||||
from nextcloud_mcp_server.observability.tracing import trace_operation
|
||||
|
||||
from .base import DocumentProcessor, ProcessingResult, ProcessorError
|
||||
from .classifier import classify_from_text
|
||||
from .classifier import classify_from_text, image_coverage_per_page
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -233,17 +233,39 @@ class ProcessorRegistry:
|
||||
fast, content, content_type, filename, options, progress_callback
|
||||
)
|
||||
|
||||
# Tier-0 classification from the extraction (cheap: no PDF re-open).
|
||||
# Tier-0 classification from the extraction (cheap: text-only, no PDF
|
||||
# re-open). Scan detection (image analysis, re-opens the PDF) runs only
|
||||
# when OCR + detect_scanned are enabled, so its cost is paid by
|
||||
# OCR-opted-in tenants only.
|
||||
classification = None
|
||||
if settings.document_classify_enabled and result.success:
|
||||
try:
|
||||
image_coverage = None
|
||||
if (
|
||||
settings.document_ocr_enabled
|
||||
and settings.document_ocr_detect_scanned
|
||||
):
|
||||
try:
|
||||
image_coverage = image_coverage_per_page(content)
|
||||
except Exception:
|
||||
logger.debug(
|
||||
"Scan detection failed for %s; using text-only signals",
|
||||
filename or "<bytes>",
|
||||
exc_info=True,
|
||||
)
|
||||
classification = classify_from_text(
|
||||
result.text, result.metadata.get("page_boundaries") or []
|
||||
result.text,
|
||||
result.metadata.get("page_boundaries") or [],
|
||||
min_text_quality=settings.document_ocr_min_text_quality,
|
||||
min_page_chars=settings.document_ocr_min_page_chars,
|
||||
page_fraction=settings.document_ocr_page_fraction,
|
||||
image_coverage=image_coverage,
|
||||
)
|
||||
record_document_classification(
|
||||
classification.recommended_tier,
|
||||
classification.flags,
|
||||
classification.mean_text_quality,
|
||||
classification.ocr_page_fraction,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
|
||||
@@ -299,6 +299,18 @@ document_text_quality = Histogram(
|
||||
buckets=(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0),
|
||||
)
|
||||
|
||||
# Per-document fraction of OCR-worthy pages (near-empty / junk-quality / scanned).
|
||||
# This is the value the DOCUMENT_OCR_PAGE_FRACTION threshold acts on, so its
|
||||
# distribution per tenant is the lever for tuning OCR escalation (quality vs
|
||||
# cost): how many docs sit just below/above the cutoff. Pair with
|
||||
# document_text_quality (where to set the per-page quality floor) and
|
||||
# document_escalation_total (realized OCR volume).
|
||||
document_ocr_page_fraction = Histogram(
|
||||
"astrolabe_document_ocr_page_fraction",
|
||||
"Tier-0 fraction of OCR-worthy pages per document (0=all-clean, 1=all-bad)",
|
||||
buckets=(0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0),
|
||||
)
|
||||
|
||||
# --- Embedding stages ---------------------------------------------------------
|
||||
|
||||
embedding_duration_seconds = Histogram(
|
||||
@@ -665,17 +677,23 @@ def record_document_parse_failed(reason: str) -> None:
|
||||
|
||||
|
||||
def record_document_classification(
|
||||
recommended_tier: str, flags: set[str], mean_text_quality: float
|
||||
recommended_tier: str,
|
||||
flags: set[str],
|
||||
mean_text_quality: float,
|
||||
ocr_page_fraction: float = 0.0,
|
||||
) -> None:
|
||||
"""Record a tier-0 classification result (shadow mode -- observability only).
|
||||
"""Record a tier-0 classification result.
|
||||
|
||||
Primitive args (not the DocClassification object) keep the observability
|
||||
layer free of a dependency on document_processors.
|
||||
layer free of a dependency on document_processors. ``mean_text_quality`` and
|
||||
``ocr_page_fraction`` feed the two histograms operators use to tune the OCR
|
||||
escalation thresholds per tenant (quality vs cost).
|
||||
"""
|
||||
document_classified_total.labels(recommended_tier=recommended_tier).inc()
|
||||
for flag in flags:
|
||||
document_classifier_flag_total.labels(flag=flag).inc()
|
||||
document_text_quality.observe(mean_text_quality)
|
||||
document_ocr_page_fraction.observe(ocr_page_fraction)
|
||||
|
||||
|
||||
def record_embedding(
|
||||
|
||||
Reference in New Issue
Block a user