feat: tier-0 document classifier in shadow mode
First step of the tiered document-processor effort (Deck #203): a cheap, local pre-pass that recommends which extraction tier a PDF should start in, emitting metrics WITHOUT changing routing yet -- so we gather per-tenant doc-mix data before turning escalation on. document_processors/classifier.py: classify_pdf(content) -> DocClassification. Page-sampled (bounded on large docs), <~1s. Cheap signals only -- text-layer chars, a text-quality score (catches the "Student 147" failure where a text layer exists but is mashed/space-less junk), and image coverage. A page that is mostly a raster image routes to OCR: its content (handwriting, stamps) isn't in any text layer. Deliberately no get_drawings/graphics-density signal -- it's slow on the exact pages it'd flag, the hotfix's graphics_limit already makes the parse safe, and the (future) tier-1 quality gate catches lost tables. Validated on the sample corpus: born-digital 2-col arxiv and a digital student record -> fast (tier 1); a scanned+handwritten form -> ocr (tier 3). Wiring (vector/processor.py): _shadow_classify runs the classifier on PDFs in a worker thread, best-effort (never blocks/fails indexing), gated by the new DOCUMENT_CLASSIFY_ENABLED setting. Metrics: astrolabe_document_classified_total {recommended_tier}, astrolabe_document_classifier_flag_total{flag}, astrolabe_document_text_quality histogram. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
dd335275ac
commit
044c1da750
@@ -17,10 +17,12 @@ from nextcloud_mcp_server.acl_hash import compute_acl_hash
|
||||
from nextcloud_mcp_server.client import NextcloudClient
|
||||
from nextcloud_mcp_server.config import get_settings
|
||||
from nextcloud_mcp_server.document_processors import get_registry
|
||||
from nextcloud_mcp_server.document_processors.classifier import classify_pdf
|
||||
from nextcloud_mcp_server.embedding import get_bm25_service, get_embedding_service
|
||||
from nextcloud_mcp_server.models.deck import DeckCard
|
||||
from nextcloud_mcp_server.observability.metrics import (
|
||||
record_document_chunks,
|
||||
record_document_classification,
|
||||
record_document_parse_failed,
|
||||
record_embedding,
|
||||
record_qdrant_operation,
|
||||
@@ -164,6 +166,29 @@ async def processor_task(
|
||||
logger.info("Processor %s stopped", worker_id)
|
||||
|
||||
|
||||
async def _shadow_classify(content: bytes, content_type: str, file_path: str) -> None:
|
||||
"""Tier-0 classification in SHADOW mode: emit metrics, change no routing.
|
||||
|
||||
Best-effort and out of the indexing critical path -- it must never block or
|
||||
fail indexing. PDFs only (the classifier is PDF-specific). The cheap pre-pass
|
||||
runs in a worker thread so it doesn't stall the event loop.
|
||||
"""
|
||||
if content_type != "application/pdf":
|
||||
return
|
||||
try:
|
||||
c = await anyio.to_thread.run_sync(classify_pdf, content) # type: ignore[attr-defined]
|
||||
record_document_classification(c.recommended_tier, c.flags, c.mean_text_quality)
|
||||
logger.debug(
|
||||
"Tier-0 classified %s: tier=%s flags=%s quality=%s",
|
||||
file_path,
|
||||
c.recommended_tier,
|
||||
sorted(c.flags),
|
||||
c.mean_text_quality,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Tier-0 classification failed for %s", file_path, exc_info=True)
|
||||
|
||||
|
||||
async def process_document(
|
||||
doc_task: DocumentTask, nc_client: NextcloudClient, *, max_retries: int = 3
|
||||
):
|
||||
@@ -534,6 +559,10 @@ async def _index_document(
|
||||
"vector_sync.file_size": len(content_bytes),
|
||||
},
|
||||
):
|
||||
# Tier-0 shadow classification (observability only; no routing change).
|
||||
if settings.document_classify_enabled:
|
||||
await _shadow_classify(content_bytes, content_type, file_path)
|
||||
|
||||
# Use document processor registry to extract text
|
||||
registry = get_registry()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user