admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: quality + scan OCR escalation trigger (junk-text-layer scans)

Browse Source 
The hot-path classifier escalated to OCR purely on character count, so a
scanned/handwritten PDF with a low-quality embedded text layer (>16 chars/page
but garbled) routed `fast` and indexed the junk -- e.g. Student 147.pdf's
"Little Acoms Primary"/"0110912020", which pollutes the vector and demotes the
doc in search (Deck #207).

- classifier: recalibrate `_text_quality` with a long-token-fraction term that
  detects word-merging (dropped inter-word spaces) -- the dominant junk-layer
  failure the old whitespace/overlong(>20) terms missed. Measured: the Student
  147 scan ~0.42 (60% pages junk) vs >=0.94 for clean digital docs.
- classify_from_text now routes on quality + scan: a page is OCR-worthy if
  near-empty OR low text-quality OR (when OCR + scan detection are enabled) it's
  mostly a raster image. New `image_coverage_per_page` re-opens the PDF for the
  scan signal, so that cost is paid only by OCR-opted-in tenants. Thresholds are
  passed in from per-tenant settings (keyword-only).
- config: 4 per-tenant settings -- DOCUMENT_OCR_MIN_TEXT_QUALITY (0.5),
  DOCUMENT_OCR_PAGE_FRACTION (0.5), DOCUMENT_OCR_MIN_PAGE_CHARS (16),
  DOCUMENT_OCR_DETECT_SCANNED (true) -- with range validators.
- metrics: new astrolabe_document_ocr_page_fraction histogram (the value the
  page-fraction threshold acts on) alongside document_text_quality, so operators
  can tune the OCR escalation per tenant (quality vs cost).

Escalation gate, OCR backends, and off-by-default behavior unchanged (#858).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 04:44:15 +02:00
co-authored by Claude Opus 4.8
parent d421bf6953
commit b1f347b8fc
7 changed files with 250 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/document_processors/registry.py
+25 -3
View File
@@ -14,7 +14,7 @@ from nextcloud_mcp_server.observability.metrics import (
from nextcloud_mcp_server.observability.tracing import trace_operation
from .base import DocumentProcessor, ProcessingResult, ProcessorError
from .classifier import classify_from_text
from .classifier import classify_from_text, image_coverage_per_page
logger = logging.getLogger(__name__)
@@ -233,17 +233,39 @@ class ProcessorRegistry:
fast, content, content_type, filename, options, progress_callback
)
# Tier-0 classification from the extraction (cheap: no PDF re-open).
# Tier-0 classification from the extraction (cheap: text-only, no PDF
# re-open). Scan detection (image analysis, re-opens the PDF) runs only
# when OCR + detect_scanned are enabled, so its cost is paid by
# OCR-opted-in tenants only.
classification = None
if settings.document_classify_enabled and result.success:
try:
image_coverage = None
if (
settings.document_ocr_enabled
and settings.document_ocr_detect_scanned
):
try:
image_coverage = image_coverage_per_page(content)
except Exception:
logger.debug(
"Scan detection failed for %s; using text-only signals",
filename or "<bytes>",
exc_info=True,
)
classification = classify_from_text(
result.text, result.metadata.get("page_boundaries") or []
result.text,
result.metadata.get("page_boundaries") or [],
min_text_quality=settings.document_ocr_min_text_quality,
min_page_chars=settings.document_ocr_min_page_chars,
page_fraction=settings.document_ocr_page_fraction,
image_coverage=image_coverage,
)
record_document_classification(
classification.recommended_tier,
classification.flags,
classification.mean_text_quality,
classification.ocr_page_fraction,
)
except Exception:
logger.warning(