admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: tiered PDF processor with pypdfium2 fast path (deprecate pymupdf4llm)

Browse Source 
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205,
follows the tier-0 classifier #855). pypdfium2 becomes the default and only
hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle.

Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF
parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per
the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2
(Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout ->
0.2s) with no table-detection bomb.

- document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text +
  exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now
  tier "structured" (the rollback engine), registered but not default.
- registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then
  classification is DERIVED from that text (classifier.classify_from_text -- no
  PDF re-open), records the classification metrics, and escalates scanned /
  no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off;
  no provider yet, so fast is terminal). Wires record_document_escalation + the
  real "escalated" span attribute (was hardcoded False).
- Removes the separate _shadow_classify pass from vector/processor.py -- it
  re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated
  CPU that lowered throughput); classification now rides the tier-1 extraction.
- Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback,
  enum-validated), document_ocr_enabled (default false).

Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify
recording, OCR escalation on/off), classify_from_text. Full unit suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 01:32:14 +02:00
co-authored by Claude Opus 4.8
parent 967298ddbe
commit c48a797896
13 changed files with 608 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/document_processors/registry.py
+142 -13
View File
@@ -5,10 +5,16 @@ import time
from collections.abc import Awaitable, Callable
from typing import Any
from nextcloud_mcp_server.observability.metrics import record_document_parse
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.observability.metrics import (
record_document_classification,
record_document_escalation,
record_document_parse,
)
from nextcloud_mcp_server.observability.tracing import trace_operation
from .base import DocumentProcessor, ProcessingResult, ProcessorError
from .classifier import classify_from_text
logger = logging.getLogger(__name__)
@@ -140,7 +146,7 @@ class ProcessorRegistry:
Raises:
ProcessorError: If no processor found or processing fails
"""
# Find processor
# Forced processor bypasses tiering.
if processor_name:
processor = self.get_processor(processor_name)
if not processor:
@@ -148,14 +154,142 @@ class ProcessorRegistry:
f"Processor '{processor_name}' not found. "
f"Available: {', '.join(self.list_processors())}"
)
else:
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
# PDFs go through the tiered pipeline (tier-0 classify -> tier-1 fast ->
# tier-3 OCR escalation). Everything else uses priority selection.
if content_type.split(";")[0].strip().lower() == "application/pdf":
return await self._process_pdf(
content, content_type, filename, options, progress_callback
)
processor = self.find_processor(content_type)
if not processor:
raise ProcessorError(
f"No processor found for type: {content_type}. "
f"Registered processors: {', '.join(self.list_processors())}"
)
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
def _pdf_processor_for_tier(self, tier: str) -> DocumentProcessor | None:
"""First registered processor of ``tier`` that handles PDFs."""
for name in self._priority_order:
processor = self._processors[name][0]
if processor.tier == tier and processor.supports("application/pdf"):
return processor
return None
async def _process_pdf(
self,
content: bytes,
content_type: str,
filename: str | None,
options: dict[str, Any] | None,
progress_callback: (
Callable[[float, float | None, str | None], Awaitable[None]] | None
),
) -> ProcessingResult:
"""Tiered PDF pipeline.
pypdfium2 ``fast`` extracts first; classification is then derived from
that text (no PDF re-open), and a scanned/no-text-layer doc escalates to
the ``ocr`` tier when enabled. ``document_tier1_engine="pymupdf"`` is a
deprecated rollback that pins the structured engine instead.
"""
settings = get_settings()
if settings.document_tier1_engine == "pymupdf":
processor = self._pdf_processor_for_tier(
"structured"
) or self.find_processor(content_type)
if processor is None:
raise ProcessorError("No PDF processor registered")
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
fast = self._pdf_processor_for_tier("fast")
if fast is None:
processor = self.find_processor(content_type)
if not processor:
raise ProcessorError(
f"No processor found for type: {content_type}. "
f"Registered processors: {', '.join(self.list_processors())}"
if processor is None:
raise ProcessorError("No PDF processor registered")
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
result = await self._run_processor(
fast, content, content_type, filename, options, progress_callback
)
# Tier-0 classification from the extraction (cheap: no PDF re-open).
classification = None
if settings.document_classify_enabled and result.success:
try:
classification = classify_from_text(
result.text, result.metadata.get("page_boundaries") or []
)
record_document_classification(
classification.recommended_tier,
classification.flags,
classification.mean_text_quality,
)
except Exception:
logger.warning(
"Tier-0 classification failed for %s",
filename or "<bytes>",
exc_info=True,
)
# Escalate scanned / no-text-layer PDFs to OCR (tier-3) when enabled and
# a provider is registered. The fast tier is terminal otherwise.
if (
classification is not None
and classification.recommended_tier == "ocr"
and settings.document_ocr_enabled
):
ocr = self._pdf_processor_for_tier("ocr")
if ocr is not None:
reason = (
"empty_text"
if classification.total_chars == 0
else "low_confidence"
)
record_document_escalation("fast", "ocr", reason)
logger.info(
"Escalating %s fast->ocr (reason=%s)",
filename or "<bytes>",
reason,
)
return await self._run_processor(
ocr,
content,
content_type,
filename,
options,
progress_callback,
escalated=True,
)
return result
async def _run_processor(
self,
processor: DocumentProcessor,
content: bytes,
content_type: str,
filename: str | None = None,
options: dict[str, Any] | None = None,
progress_callback: (
Callable[[float, float | None, str | None], Awaitable[None]] | None
) = None,
*,
escalated: bool = False,
) -> ProcessingResult:
"""Run one processor with the per-processor span + parse metrics."""
tier = processor.tier
logger.info(
"Processing with '%s' processor",
@@ -167,11 +301,6 @@ class ProcessorRegistry:
},
)
# Process (instrumented: per-processor span + parse metrics).
# NOTE: when the tiered pipeline (docling/OCR/LLM) lands, escalation
# decisions are recorded here via record_document_escalation() and an
# add_span_event("document.escalation", ...) -- the escalated=False
# attribute and the metric are wired ahead of that.
byte_size = len(content)
start_time = time.time()
with trace_operation(
@@ -181,7 +310,7 @@ class ProcessorRegistry:
"processor.tier": tier,
"mime_type": content_type,
"byte_size": byte_size,
"escalated": False,
"escalated": escalated,
},
record_exception=True,
) as span: