admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: tiered PDF processor with pypdfium2 fast path (deprecate pymupdf4llm)

Browse Source 
Replaces single-engine pymupdf4llm extraction with a tiered pipeline (Deck #205,
follows the tier-0 classifier #855). pypdfium2 becomes the default and only
hot-path PDF extractor; pymupdf4llm is deprecated to a rollback toggle.

Why: pymupdf4llm's O(n^2) find_tables drove the OOM (#852) and the form-PDF
parse timeouts (#856), carries AGPL/commercial licensing liability, and -- per
the benchmarks -- recovers near-zero usable tables on the real corpus. pypdfium2
(Apache/BSD) extracts the same text far faster (Student 1a.pdf: 120s timeout ->
0.2s) with no table-detection bomb.

- document_processors/pypdfium2_fast.py: tier-1 "fast" processor emitting text +
  exact page_boundaries (the pdf_highlighter contract). pymupdf processor is now
  tier "structured" (the rollback engine), registered but not default.
- registry: tiered routing in ProcessorRegistry. tier-1 fast extracts, then
  classification is DERIVED from that text (classifier.classify_from_text -- no
  PDF re-open), records the classification metrics, and escalates scanned /
  no-text-layer docs to the "ocr" tier when document_ocr_enabled (default off;
  no provider yet, so fast is terminal). Wires record_document_escalation + the
  real "escalated" span attribute (was hardcoded False).
- Removes the separate _shadow_classify pass from vector/processor.py -- it
  re-opened every PDF and re-extracted text (~0.5-1.3s/doc of pure duplicated
  CPU that lowered throughput); classification now rides the tier-1 extraction.
- Settings: document_tier1_engine ("pypdfium2" default | "pymupdf" rollback,
  enum-validated), document_ocr_enabled (default false).

Tests: pypdfium2 extractor, registry tiering (fast routing, rollback, classify
recording, OCR escalation on/off), classify_from_text. Full unit suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-05 01:32:14 +02:00
co-authored by Claude Opus 4.8
parent 967298ddbe
commit c48a797896
13 changed files with 608 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/document_processors/__init__.py
+7 -1
View File
@@ -2,10 +2,15 @@
from .base import DocumentProcessor, ProcessingResult, ProcessorError
from .pymupdf import PyMuPDFProcessor
from .pypdfium2_fast import Pypdfium2FastProcessor
from .registry import ProcessorRegistry, get_registry
# Register processors at module initialization
# Register processors at module initialization. The tiered PDF pipeline selects
# by tier (not priority): Pypdfium2FastProcessor is the ``fast`` tier and
# PyMuPDFProcessor the ``structured`` escalation target. Priority still orders
# the non-tiered fallback path and other MIME types.
_registry = get_registry()
_registry.register(Pypdfium2FastProcessor(), priority=20)
_registry.register(PyMuPDFProcessor(), priority=10)
__all__ = [
@@ -15,4 +20,5 @@ __all__ = [
"ProcessorRegistry",
"get_registry",
"PyMuPDFProcessor",
"Pypdfium2FastProcessor",
]
nextcloud_mcp_server/document_processors/classifier.py
+70 -19
View File
@@ -1,32 +1,31 @@
"""Tier-0 document classifier.
A cheap (<~1s), local pre-pass over a PDF that decides which extraction tier a
document should start in, BEFORE the expensive parse. It runs in *shadow mode*
first: emit the signals as metrics, change no routing, and gather per-tenant
data to tune the thresholds.
Decides which extraction tier a PDF should escalate to, from cheap signals:
* text_quality -- is the text layer usable, or mashed/space-less junk? (the
"Student 147" lesson: a text layer can exist yet be unusable, e.g.
"01322234567mobile")
* image_coverage -- a page that is mostly a raster image is a scan/photo whose
content isn't fully in any text layer.
* no text layer -- the strongest OCR signal available from text alone.
Signals (all cheap; no get_drawings, which is itself slow on the graphics-heavy
pages we'd want to flag -- the parse-time ``graphics_limit`` already makes those
safe, and the tier-1 quality gate catches unrecovered tables post-extraction):
Two entry points:
* ``classify_from_text(text, page_boundaries)`` -- the HOT PATH. Derives the
text-quality/no-text-layer signal from the text the registry's tier-1 step
already extracted, so it adds ~no cost. No image analysis.
* ``classify_pdf(content)`` -- a standalone/diagnostic pass that re-opens the
PDF and adds image-coverage analysis. More expensive; used off the hot path.
* text_layer_chars -- extractable text per page
* text_quality -- is the text layer usable, or mashed/space-less junk?
(the "Student 147" lesson: a text layer can exist yet
be unusable, e.g. "01322234567mobile")
* image_coverage -- fraction of the page covered by raster images
(full-page image + poor text => scanned)
Recommended tier:
* ``ocr`` -- scanned / no-usable-text-layer (route to tier 3, when enabled)
* ``fast`` -- a usable digital text layer (stay on tier 1)
From these it picks a recommended starting tier:
* ``ocr`` -- scanned / image-only / bad-text-layer (route to tier 3)
* ``fast`` -- a usable digital text layer (route to tier 1)
``structured`` (tier 2 / docling) is intentionally not produced here -- that tier
is a separate service and is reached via the tier-1 quality gate, not tier-0.
``structured`` (tier 2 / docling) is a separate service, not produced here.
"""
import logging
import re
from dataclasses import dataclass, field
from typing import Any
logger = logging.getLogger(__name__)
@@ -40,6 +39,8 @@ IMAGE_COVERAGE_SCANNED = 0.80
MIN_TEXT_QUALITY = 0.45
# Fraction of sampled pages that must look scanned/bad for a doc->ocr verdict.
OCR_PAGE_FRACTION = 0.5
# A page with fewer extracted chars than this has effectively no text layer.
MIN_PAGE_CHARS = 16
_WORD_RE = re.compile(r"\S+")
@@ -177,3 +178,53 @@ def classify_pdf(content: bytes) -> DocClassification:
flags=flags,
pages=pages,
)
def classify_from_text(
full_text: str, page_boundaries: list[dict[str, Any]]
) -> DocClassification:
"""Classify from text already extracted by tier-1 -- no PDF re-open.
The hot-path classifier: it derives the text-quality signal from the
extraction the registry already ran, so it adds ~no cost (vs ``classify_pdf``,
which re-opens the PDF and re-extracts). It does NOT do image analysis, so it
cannot distinguish a scanned-with-text-layer page (that needs the image pass,
which only matters once OCR routing is enabled). A page with effectively no
text layer is the one OCR-worthy signal available from text alone.
``page_boundaries`` are ``{page, start_offset, end_offset}`` indexing into
``full_text`` (the tier-1/pdf_highlighter contract).
"""
pages: list[PageSignals] = []
for b in page_boundaries:
seg = full_text[b["start_offset"] : b["end_offset"]]
needs_ocr = len(seg.strip()) < MIN_PAGE_CHARS
pages.append(
PageSignals(b["page"], len(seg), 0.0, _text_quality(seg), needs_ocr)
)
sampled = len(pages)
total_chars = sum(p.char_count for p in pages)
mean_quality = (
round(sum(p.text_quality for p in pages) / sampled, 3) if sampled else 0.0
)
ocr_frac = (sum(p.needs_ocr for p in pages) / sampled) if sampled else 1.0
flags: set[str] = set()
if total_chars == 0:
flags.add("no_text_layer")
elif mean_quality < MIN_TEXT_QUALITY:
flags.add("bad_text_layer")
recommended = "ocr" if ocr_frac >= OCR_PAGE_FRACTION else "fast"
return DocClassification(
page_count=len(page_boundaries),
sampled_pages=sampled,
total_chars=total_chars,
mean_text_quality=mean_quality,
ocr_page_fraction=round(ocr_frac, 3),
recommended_tier=recommended,
flags=flags,
pages=pages,
)
nextcloud_mcp_server/document_processors/pymupdf.py
+7
View File
@@ -73,6 +73,13 @@ class PyMuPDFProcessor(DocumentProcessor):
def name(self) -> str:
return "pymupdf"
@property
def tier(self) -> str:
# pymupdf4llm recovers markdown structure (headings, lists, tables) via
# the expensive graphics-limited table detection -- it is the
# ``structured`` escalation target above the pypdfium2 ``fast`` tier.
return "structured"
@property
def supported_mime_types(self) -> set[str]:
return self.SUPPORTED_TYPES
nextcloud_mcp_server/document_processors/pypdfium2_fast.py
+123
View File
@@ -0,0 +1,123 @@
"""Tier-1 fast PDF text extractor (pypdfium2).
A permissively-licensed (Apache/BSD-2) fast path that extracts a PDF's text
layer + page boundaries WITHOUT pymupdf4llm's expensive O(n^2) table/graphics
analysis. For born-digital PDFs (the tier-0 classifier's ``fast`` verdict) this
returns clean text in well under a second -- including the form/table PDFs that
timed out under pymupdf4llm (e.g. ``Student 1a.pdf``: 120s timeout -> ~1s here).
bbox is re-derived from the PDF bytes + ``page_boundaries`` by
``search/pdf_highlighter``, so this processor only needs to emit ``text`` and
``metadata["page_boundaries"]`` for chunk highlighting to keep working.
It deliberately does NOT recover tables/layout; a low-quality result is meant to
escalate to the ``structured`` tier (pymupdf4llm, graphics_limit-guarded) via the
registry (B2 escalation wiring).
"""
import logging
from collections.abc import Awaitable, Callable
from typing import Any
import anyio
from .base import DocumentProcessor, ProcessingResult
logger = logging.getLogger(__name__)
def _extract(content: bytes) -> tuple[str, dict[str, Any]]:
"""Extract concatenated text + metadata from a PDF (runs in a worker thread).
``page_boundaries`` offsets index into the returned text, which is the page
texts joined with no separator so the offsets stay exact (the contract
``search/pdf_highlighter`` and the chunker rely on).
"""
import pypdfium2 as pdfium # noqa: PLC0415 -- keep the native import lazy
pdf = pdfium.PdfDocument(content)
try:
page_texts: list[str] = []
for i in range(len(pdf)):
page = pdf[i]
textpage = page.get_textpage()
try:
page_texts.append(textpage.get_text_bounded() or "")
finally:
textpage.close()
page.close()
doc_meta = pdf.get_metadata_dict() or {}
finally:
pdf.close()
page_boundaries: list[dict[str, Any]] = []
offset = 0
for n, text in enumerate(page_texts, start=1):
page_boundaries.append(
{"page": n, "start_offset": offset, "end_offset": offset + len(text)}
)
offset += len(text)
full_text = "".join(page_texts)
metadata: dict[str, Any] = {
"page_count": len(page_texts),
"page_boundaries": page_boundaries,
}
title = doc_meta.get("Title")
if title:
metadata["title"] = title
return full_text, metadata
class Pypdfium2FastProcessor(DocumentProcessor):
"""Tier-1 fast PDF text extractor backed by pypdfium2."""
@property
def name(self) -> str:
return "pypdfium2_fast"
@property
def tier(self) -> str:
return "fast"
@property
def supported_mime_types(self) -> set[str]:
return {"application/pdf"}
async def process(
self,
content: bytes,
content_type: str,
filename: str | None = None,
options: dict[str, Any] | None = None,
progress_callback: (
Callable[[float, float | None, str | None], Awaitable[None]] | None
) = None,
) -> ProcessingResult:
if progress_callback:
await progress_callback(0, 100, "Extracting text (pypdfium2)")
try:
full_text, metadata = await anyio.to_thread.run_sync( # type: ignore[attr-defined]
_extract, content
)
except Exception as e:
# Fast path is best-effort: a failure here escalates rather than
# crashing the pipeline. pypdfium2 has no O(n^2) bomb, so this is a
# genuinely malformed PDF, not a resource blowup.
logger.warning(
"pypdfium2 fast extract failed for %s: %s", filename or "<bytes>", e
)
return ProcessingResult(
text="",
metadata={"parse_failed_reason": "error"},
processor=self.name,
success=False,
error=f"{type(e).__name__}: {e}",
)
metadata["file_size"] = len(content)
if progress_callback:
await progress_callback(100, 100, "Done")
return ProcessingResult(text=full_text, metadata=metadata, processor=self.name)
async def health_check(self) -> bool:
return True
nextcloud_mcp_server/document_processors/registry.py
+142 -13
View File
@@ -5,10 +5,16 @@ import time
from collections.abc import Awaitable, Callable
from typing import Any
from nextcloud_mcp_server.observability.metrics import record_document_parse
from nextcloud_mcp_server.config import get_settings
from nextcloud_mcp_server.observability.metrics import (
record_document_classification,
record_document_escalation,
record_document_parse,
)
from nextcloud_mcp_server.observability.tracing import trace_operation
from .base import DocumentProcessor, ProcessingResult, ProcessorError
from .classifier import classify_from_text
logger = logging.getLogger(__name__)
@@ -140,7 +146,7 @@ class ProcessorRegistry:
Raises:
ProcessorError: If no processor found or processing fails
"""
# Find processor
# Forced processor bypasses tiering.
if processor_name:
processor = self.get_processor(processor_name)
if not processor:
@@ -148,14 +154,142 @@ class ProcessorRegistry:
f"Processor '{processor_name}' not found. "
f"Available: {', '.join(self.list_processors())}"
)
else:
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
# PDFs go through the tiered pipeline (tier-0 classify -> tier-1 fast ->
# tier-3 OCR escalation). Everything else uses priority selection.
if content_type.split(";")[0].strip().lower() == "application/pdf":
return await self._process_pdf(
content, content_type, filename, options, progress_callback
)
processor = self.find_processor(content_type)
if not processor:
raise ProcessorError(
f"No processor found for type: {content_type}. "
f"Registered processors: {', '.join(self.list_processors())}"
)
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
def _pdf_processor_for_tier(self, tier: str) -> DocumentProcessor | None:
"""First registered processor of ``tier`` that handles PDFs."""
for name in self._priority_order:
processor = self._processors[name][0]
if processor.tier == tier and processor.supports("application/pdf"):
return processor
return None
async def _process_pdf(
self,
content: bytes,
content_type: str,
filename: str | None,
options: dict[str, Any] | None,
progress_callback: (
Callable[[float, float | None, str | None], Awaitable[None]] | None
),
) -> ProcessingResult:
"""Tiered PDF pipeline.
pypdfium2 ``fast`` extracts first; classification is then derived from
that text (no PDF re-open), and a scanned/no-text-layer doc escalates to
the ``ocr`` tier when enabled. ``document_tier1_engine="pymupdf"`` is a
deprecated rollback that pins the structured engine instead.
"""
settings = get_settings()
if settings.document_tier1_engine == "pymupdf":
processor = self._pdf_processor_for_tier(
"structured"
) or self.find_processor(content_type)
if processor is None:
raise ProcessorError("No PDF processor registered")
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
fast = self._pdf_processor_for_tier("fast")
if fast is None:
processor = self.find_processor(content_type)
if not processor:
raise ProcessorError(
f"No processor found for type: {content_type}. "
f"Registered processors: {', '.join(self.list_processors())}"
if processor is None:
raise ProcessorError("No PDF processor registered")
return await self._run_processor(
processor, content, content_type, filename, options, progress_callback
)
result = await self._run_processor(
fast, content, content_type, filename, options, progress_callback
)
# Tier-0 classification from the extraction (cheap: no PDF re-open).
classification = None
if settings.document_classify_enabled and result.success:
try:
classification = classify_from_text(
result.text, result.metadata.get("page_boundaries") or []
)
record_document_classification(
classification.recommended_tier,
classification.flags,
classification.mean_text_quality,
)
except Exception:
logger.warning(
"Tier-0 classification failed for %s",
filename or "<bytes>",
exc_info=True,
)
# Escalate scanned / no-text-layer PDFs to OCR (tier-3) when enabled and
# a provider is registered. The fast tier is terminal otherwise.
if (
classification is not None
and classification.recommended_tier == "ocr"
and settings.document_ocr_enabled
):
ocr = self._pdf_processor_for_tier("ocr")
if ocr is not None:
reason = (
"empty_text"
if classification.total_chars == 0
else "low_confidence"
)
record_document_escalation("fast", "ocr", reason)
logger.info(
"Escalating %s fast->ocr (reason=%s)",
filename or "<bytes>",
reason,
)
return await self._run_processor(
ocr,
content,
content_type,
filename,
options,
progress_callback,
escalated=True,
)
return result
async def _run_processor(
self,
processor: DocumentProcessor,
content: bytes,
content_type: str,
filename: str | None = None,
options: dict[str, Any] | None = None,
progress_callback: (
Callable[[float, float | None, str | None], Awaitable[None]] | None
) = None,
*,
escalated: bool = False,
) -> ProcessingResult:
"""Run one processor with the per-processor span + parse metrics."""
tier = processor.tier
logger.info(
"Processing with '%s' processor",
@@ -167,11 +301,6 @@ class ProcessorRegistry:
},
)
# Process (instrumented: per-processor span + parse metrics).
# NOTE: when the tiered pipeline (docling/OCR/LLM) lands, escalation
# decisions are recorded here via record_document_escalation() and an
# add_span_event("document.escalation", ...) -- the escalated=False
# attribute and the metric are wired ahead of that.
byte_size = len(content)
start_time = time.time()
with trace_operation(
@@ -181,7 +310,7 @@ class ProcessorRegistry:
"processor.tier": tier,
"mime_type": content_type,
"byte_size": byte_size,
"escalated": False,
"escalated": escalated,
},
record_exception=True,
) as span: