feat: tier-3 OCR processor (gateway or direct Mistral)
Adds the OCR escalation target the tiered registry already routes to. Scanned /
no-text-layer PDFs (the tier-0 "ocr" verdict) escalate here when
document_ocr_enabled (default off).
Two interchangeable backends, selected by document_ocr_provider
(auto | gateway | mistral | none):
- gateway: POST to the Astrolabe Cloud model gateway's /v1/ocr -- the same
M2M-authenticated gateway as embeddings, so NO provider keys live in the pod
(the platform default; reuses EMBEDDING_GATEWAY_URL + the M2M creds).
- mistral: call the Mistral OCR API directly from the pod (MISTRAL_API_KEY), for
self-hosters / deployments without the gateway.
"auto" prefers the gateway, then direct Mistral.
Both return per-page markdown joined into text + exact page_boundaries (the
pdf_highlighter contract; bbox re-derived from the PDF bytes as for other tiers).
Validated end-to-end via direct Mistral on the scanned Student 147.pdf:
success, 15 pages, 22k chars, offsets exact, ~4s.
Settings: document_ocr_provider (enum-validated), document_ocr_model
("mistral/mistral-ocr-latest" -- gateway routes on the prefix, the direct mistral
backend strips it). OcrProcessor registered at lowest priority so it is never the
non-tiered default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
c48a797896
commit
3bd1b46d9c
@@ -143,6 +143,12 @@ _DEFAULTS: dict[str, Any] = {
|
||||
# escalation target and is off by default (no provider wired yet).
|
||||
"document_tier1_engine": "pypdfium2",
|
||||
"document_ocr_enabled": False,
|
||||
# OCR backend: "auto" picks gateway (if EMBEDDING_GATEWAY_URL) else mistral
|
||||
# (if MISTRAL_API_KEY); "gateway"/"mistral" force one; "none" disables.
|
||||
"document_ocr_provider": "auto",
|
||||
# Provider-namespaced OCR model id (gateway routes on the prefix; the direct
|
||||
# mistral backend strips it).
|
||||
"document_ocr_model": "mistral/mistral-ocr-latest",
|
||||
# Observability
|
||||
"metrics_enabled": True,
|
||||
"metrics_port": 9090,
|
||||
@@ -298,6 +304,9 @@ _dynaconf = Dynaconf(
|
||||
# Enum constraints
|
||||
Validator("LOG_FORMAT", is_in=["text", "json"]),
|
||||
Validator("DOCUMENT_TIER1_ENGINE", is_in=["pypdfium2", "pymupdf"]),
|
||||
Validator(
|
||||
"DOCUMENT_OCR_PROVIDER", is_in=["auto", "gateway", "mistral", "none"]
|
||||
),
|
||||
Validator(
|
||||
"LOG_LEVEL",
|
||||
is_in=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
|
||||
@@ -742,9 +751,15 @@ class Settings:
|
||||
# permissive license, no find_tables) is the hot path; "pymupdf" is a
|
||||
# deprecated rollback to pymupdf4llm (AGPL, graphics-limited) for one corpus.
|
||||
document_tier1_engine: str = "pypdfium2"
|
||||
# Route scanned/no-text-layer PDFs to the tier-3 OCR provider. Off until an
|
||||
# OCR backend is wired; when off, the fast tier is terminal.
|
||||
# Route scanned/no-text-layer PDFs to the tier-3 OCR provider. Off by
|
||||
# default; when off, the fast tier is terminal.
|
||||
document_ocr_enabled: bool = False
|
||||
# OCR backend selection: "auto" | "gateway" | "mistral" | "none".
|
||||
document_ocr_provider: str = "auto"
|
||||
# Provider-namespaced OCR model id (e.g. "mistral/mistral-ocr-latest"). The
|
||||
# gateway routes on the "<provider>/" prefix; the direct mistral backend
|
||||
# strips it.
|
||||
document_ocr_model: str = "mistral/mistral-ocr-latest"
|
||||
|
||||
# Observability settings
|
||||
metrics_enabled: bool = True
|
||||
@@ -1360,6 +1375,8 @@ def get_settings() -> Settings:
|
||||
"document_classify_enabled": "DOCUMENT_CLASSIFY_ENABLED",
|
||||
"document_tier1_engine": "DOCUMENT_TIER1_ENGINE",
|
||||
"document_ocr_enabled": "DOCUMENT_OCR_ENABLED",
|
||||
"document_ocr_provider": "DOCUMENT_OCR_PROVIDER",
|
||||
"document_ocr_model": "DOCUMENT_OCR_MODEL",
|
||||
# Observability settings
|
||||
"metrics_enabled": "METRICS_ENABLED",
|
||||
"metrics_port": "METRICS_PORT",
|
||||
|
||||
Reference in New Issue
Block a user