feat: tier-0 document classifier in shadow mode
First step of the tiered document-processor effort (Deck #203): a cheap, local pre-pass that recommends which extraction tier a PDF should start in, emitting metrics WITHOUT changing routing yet -- so we gather per-tenant doc-mix data before turning escalation on. document_processors/classifier.py: classify_pdf(content) -> DocClassification. Page-sampled (bounded on large docs), <~1s. Cheap signals only -- text-layer chars, a text-quality score (catches the "Student 147" failure where a text layer exists but is mashed/space-less junk), and image coverage. A page that is mostly a raster image routes to OCR: its content (handwriting, stamps) isn't in any text layer. Deliberately no get_drawings/graphics-density signal -- it's slow on the exact pages it'd flag, the hotfix's graphics_limit already makes the parse safe, and the (future) tier-1 quality gate catches lost tables. Validated on the sample corpus: born-digital 2-col arxiv and a digital student record -> fast (tier 1); a scanned+handwritten form -> ocr (tier 3). Wiring (vector/processor.py): _shadow_classify runs the classifier on PDFs in a worker thread, best-effort (never blocks/fails indexing), gated by the new DOCUMENT_CLASSIFY_ENABLED setting. Metrics: astrolabe_document_classified_total {recommended_tier}, astrolabe_document_classifier_flag_total{flag}, astrolabe_document_text_quality histogram. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
dd335275ac
commit
044c1da750
@@ -136,6 +136,8 @@ _DEFAULTS: dict[str, Any] = {
|
||||
"document_pdf_graphics_limit": 1000,
|
||||
"document_parse_timeout_seconds": 120.0,
|
||||
"document_parse_mem_limit_mb": 1536,
|
||||
# Tier-0 classifier (shadow mode: emits metrics, no routing change)
|
||||
"document_classify_enabled": True,
|
||||
# Observability
|
||||
"metrics_enabled": True,
|
||||
"metrics_port": 9090,
|
||||
@@ -727,6 +729,9 @@ class Settings:
|
||||
# RLIMIT_AS in the parse subprocess (below the pod limit). Applied once per
|
||||
# worker for its lifetime, so changing it needs a pod restart.
|
||||
document_parse_mem_limit_mb: int = 1536
|
||||
# Tier-0 classifier. Shadow mode for now: runs a cheap pre-pass over each PDF
|
||||
# and emits classification metrics, but does NOT change routing yet.
|
||||
document_classify_enabled: bool = True
|
||||
|
||||
# Observability settings
|
||||
metrics_enabled: bool = True
|
||||
@@ -1339,6 +1344,7 @@ def get_settings() -> Settings:
|
||||
"document_pdf_graphics_limit": "DOCUMENT_PDF_GRAPHICS_LIMIT",
|
||||
"document_parse_timeout_seconds": "DOCUMENT_PARSE_TIMEOUT_SECONDS",
|
||||
"document_parse_mem_limit_mb": "DOCUMENT_PARSE_MEM_LIMIT_MB",
|
||||
"document_classify_enabled": "DOCUMENT_CLASSIFY_ENABLED",
|
||||
# Observability settings
|
||||
"metrics_enabled": "METRICS_ENABLED",
|
||||
"metrics_port": "METRICS_PORT",
|
||||
|
||||
Reference in New Issue
Block a user