Chris CoutinhoandClaude Opus 4.8
c21804fbbc
Insert a configurable in-cluster OCR rung into the escalation ladder (Deck #353): a tier2-eligible doc is OCR'd on the on-demand burst GPU before falling through to paid upstream OCR. The in-cluster backend is reached ONLY via the embedding gateway (model prefix routes to the GPU over the tailnet) and is a config value (default surya/surya-ocr-2, swappable to e.g. lightonocr) — never hard-coded. Ladder: fast -> structured -> ocr-incluster -> ocr-upstream (queues ingest-ocr-incluster / ingest-ocr-upstream). - escalation.py: 4-tier ladder; in-cluster flag folded into the dead-letter signature. - ocr.py: OcrProcessor(name, tier, model_setting, gateway_only); build_ocr_backend( ..., model=, gateway_only=) — gateway_only forces the gateway backend (never the direct Mistral fallback), disabling the tier with a warning if no gateway URL. - registry.py: per-rung enable map; scanned docs target minimum="ocr-incluster"; inline path runs the cheapest available OCR rung. - procrastinate.py: two OCR queues; legacy ingest-ocr kept as a drain target. - config.py: DOCUMENT_OCR_INCLUSTER_ENABLED (off) + DOCUMENT_OCR_INCLUSTER_MODEL. - __init__.py: register the two OCR instances; vector/processor.py: pages_ocr metered for the upstream (paid) rung only; cli.py: new --tier choices + legacy drain. - metrics.py: zero the legacy ingest-ocr queue gauge during rollout. - tests: migrated to the split ladder + new tests (gateway-only forcing, per-tier model incl. lightonocr override, no-hard-coded-surya guard). 1792 pass; ruff + ty green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
48 lines
1.6 KiB
Python
48 lines
1.6 KiB
Python
"""Document processing plugins for extracting text from various file formats."""
|
|
|
|
from .base import DocumentProcessor, ProcessingResult, ProcessorError
|
|
from .ocr import OcrProcessor
|
|
from .pymupdf import PyMuPDFProcessor
|
|
from .pypdfium2_fast import Pypdfium2FastProcessor
|
|
from .registry import ProcessorRegistry, get_registry
|
|
|
|
# Register processors at module initialization. The tiered PDF pipeline selects
|
|
# by tier (not priority): Pypdfium2FastProcessor is the ``fast`` tier,
|
|
# PyMuPDFProcessor the ``structured`` rollback, and TWO OcrProcessor instances are
|
|
# the OCR rungs — ``ocr-incluster`` (the on-demand burst GPU, gateway-only, reached
|
|
# via the embedding gateway over the tailnet; e.g. surya) tried before
|
|
# ``ocr-upstream`` (paid Mistral). Each is reached only when its own opt-in flag is
|
|
# set. OCR gets the lowest priorities so it's never the non-tiered default for PDFs.
|
|
_registry = get_registry()
|
|
_registry.register(Pypdfium2FastProcessor(), priority=20)
|
|
_registry.register(PyMuPDFProcessor(), priority=10)
|
|
_registry.register(
|
|
OcrProcessor(
|
|
name="ocr-incluster",
|
|
tier="ocr-incluster",
|
|
model_setting="document_ocr_incluster_model",
|
|
gateway_only=True,
|
|
),
|
|
priority=2,
|
|
)
|
|
_registry.register(
|
|
OcrProcessor(
|
|
name="ocr-upstream",
|
|
tier="ocr-upstream",
|
|
model_setting="document_ocr_model",
|
|
gateway_only=False,
|
|
),
|
|
priority=1,
|
|
)
|
|
|
|
__all__ = [
|
|
"DocumentProcessor",
|
|
"ProcessingResult",
|
|
"ProcessorError",
|
|
"ProcessorRegistry",
|
|
"get_registry",
|
|
"PyMuPDFProcessor",
|
|
"Pypdfium2FastProcessor",
|
|
"OcrProcessor",
|
|
]
|