admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ingest): per-tier escalation via procrastinate queue-hop

Browse Source 
Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.

- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
  (reuses classify_from_text) + next_available_tier; shared _classify_result
  and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
  before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
  (queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
  task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
  counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS

INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.

Deck #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-13 13:22:18 +02:00
co-authored by Claude Opus 4.8
parent 6fab0e2ae3
commit 9676bb3106
17 changed files with 1259 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/vector/processor.py
+169 -14
View File
@@ -6,7 +6,7 @@ Processes documents from stream: fetches content, generates embeddings, stores i
import logging
import time
import uuid
from typing import Any, cast
from typing import TYPE_CHECKING, Any, cast
import anyio
import httpx
@@ -14,6 +14,12 @@ from anyio.abc import TaskStatus
from anyio.streams.memory import MemoryObjectReceiveStream
from qdrant_client.models import PointStruct
if TYPE_CHECKING:
# Type-only: the document stack is heavy (pymupdf/_isolation) and must stay
# off processor.py's import path (#877); the runtime import is lazy.
from nextcloud_mcp_server.document_processors.base import ProcessingResult
from nextcloud_mcp_server.document_processors.registry import ProcessorRegistry
from nextcloud_mcp_server.acl_hash import compute_acl_hash
from nextcloud_mcp_server.client import NextcloudClient
from nextcloud_mcp_server.config import get_settings
@@ -21,6 +27,7 @@ from nextcloud_mcp_server.embedding import get_bm25_service, get_embedding_servi
from nextcloud_mcp_server.models.deck import DeckCard
from nextcloud_mcp_server.observability.metrics import (
record_document_chunks,
record_document_escalation,
record_document_parse_failed,
record_embedding,
record_embedding_tokens,
@@ -106,6 +113,59 @@ def _drop_reason(exc: BaseException) -> str:
return "other"
def _is_pdf(content_type: str) -> bool:
"""Whether a MIME type is a PDF (parameter-tolerant)."""
return content_type.split(";")[0].strip().lower() == "application/pdf"
async def _parse_pdf_tier(
registry: "ProcessorRegistry",
content: bytes,
content_type: str,
filename: str | None,
tier: str,
settings: Any,
) -> "ProcessingResult":
"""Run a single extraction tier and apply the post-parse escalation gate.
The external per-tier ingest path (Deck #323): the procrastinate worker for
``tier`` parses with exactly that tier, then either returns the result to
index or raises ``EscalateError`` to hand the document to the next tier's
queue (the queue's retry strategy turns the raise into a native queue-hop).
The escalation metric is recorded here, at the decision point.
A hard parse failure (``result.success`` False) is returned as-is, not
escalated -- a corrupt/encrypted/oversize PDF that one engine can't open
usually defeats the others too; the caller marks it failed. This preserves
the "OCR is an enhancement, never worse than off" invariant: a tenant who has
not enabled a higher tier (or has no processor for it) simply indexes the
cheap tier's output.
"""
# Lazy import: keep the document stack (pymupdf/_isolation) off the module
# load path; this runs only on the per-tier worker, which needs it anyway.
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
EscalateError,
)
result = await registry.process_tier(content, content_type, filename, tier)
if result.success:
decision = registry.evaluate_escalation(
result, content, tier, settings, filename=filename
)
if decision is not None:
to_tier, reason = decision
record_document_escalation(tier, to_tier, reason)
logger.info(
"Escalating %s %s->%s (reason=%s)",
filename or "<bytes>",
tier,
to_tier,
reason,
)
raise EscalateError(from_tier=tier, to_tier=to_tier, reason=reason)
return result
def assign_page_numbers(chunks, page_boundaries):
"""Assign page numbers to chunks based on page boundaries.
@@ -173,6 +233,7 @@ async def record_indexing_usage(
token_count: int,
total_chars: int,
page_count: int | None,
pipeline_tier: str | None = None,
) -> None:
"""Record the billable usage events for one embedded document.
@@ -213,6 +274,11 @@ async def record_indexing_usage(
"doc_type": doc_type,
"user_id": user_id,
"total_chars": total_chars,
# Which extraction tier produced the parsed pages (Deck #323). Carried so
# the CP rollup / a future per-tier price can attribute parsing cost to
# the tier that incurred it (paid OCR vs CPU-cheap fast). None for text
# doc types, which are never parsed.
"pipeline_tier": pipeline_tier,
}
try:
store = await UsageEventStore.shared()
@@ -242,6 +308,18 @@ async def record_indexing_usage(
metadata=metadata,
enabled=True,
)
# Paid-OCR pages are metered as a SEPARATE line (Deck #323) so the
# expensive tier's cost is billable independently of CPU-cheap parsing
# -- pages_embedded counts all parsed pages, pages_ocr only the OCR
# tier's. Gated on the tier so it's emitted exactly when the doc was
# actually OCR'd; the same page_count guard above applies.
if pipeline_tier == "ocr":
await store.record_usage_event(
metric="pages_ocr",
value=page_count,
metadata=metadata,
enabled=True,
)
except Exception:
# Reached only when shared()/store construction itself raises
# (record_usage_event swallows its own write failures). Metering is on,
@@ -393,7 +471,11 @@ async def _reconcile_tag_event(
async def process_document(
doc_task: DocumentTask, nc_client: NextcloudClient, *, max_retries: int = 3
doc_task: DocumentTask,
nc_client: NextcloudClient,
*,
max_retries: int = 3,
tier: str | None = None,
):
"""
Process a single document: fetch, tokenize, embed, store in Qdrant.
@@ -407,6 +489,11 @@ async def process_document(
(3) suits the in-process SQLite pool, which has no durable retry. The
procrastinate worker passes ``1`` so durable retry is owned by the
queue (and survives worker crashes), avoiding compounding 3×N retries.
tier: Extraction tier to run for PDFs on the external per-tier path (Deck
#323) -- the procrastinate worker passes the tier matching its queue.
``None`` (the default, used by the in-process/memory pool) runs the
inline tiered pipeline (``registry.process``: fast -> OCR escalation
in one call) and never raises ``EscalateError``.
Retry layering: the embedding provider adds its own transient retry (5
attempts, 2s→60s backoff — card 309) *inside* each of these attempts. On the
@@ -415,6 +502,19 @@ async def process_document(
re-picked on the next scan; the procrastinate path (max_retries=1) caps it at
one outer attempt (~30s) and defers. Don't stack a third retry layer here.
"""
# EscalateError is a control-flow signal that arises ONLY on the per-tier
# external path (tier set). Bind the class lazily there so the in-process /
# memory path never pulls the document stack at call time (mirrors the lazy
# get_registry import; see #877). When tier is None it can't be raised, so
# the guards below stay inert.
escalate_error_cls: type[BaseException] | None = None
if tier is not None:
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
EscalateError,
)
escalate_error_cls = EscalateError
start_time = time.time()
logger.debug(
@@ -484,7 +584,9 @@ async def process_document(
for attempt in range(max_retries):
try:
indexed = await _index_document(doc_task, nc_client, qdrant_client)
indexed = await _index_document(
doc_task, nc_client, qdrant_client, tier=tier
)
# A permanent parse failure returns False: it was already
# recorded (document_parse_failed_total + the registry's
@@ -506,6 +608,14 @@ async def process_document(
return # Success
except Exception as e:
# An escalation signal is control flow, not a failure:
# propagate it untouched so the procrastinate retry strategy
# can hop the job to the next tier's queue. Never retry it
# in-process and never count it as a drop.
if escalate_error_cls is not None and isinstance(
e, escalate_error_cls
):
raise
if attempt < max_retries - 1:
logger.warning(
"Retry %s/%s for %s_%s: %s",
@@ -556,7 +666,12 @@ async def process_document(
record_ingest_dropped(reason)
raise
except Exception:
except Exception as e:
# An escalation signal must reach the procrastinate retry strategy
# un-recorded -- it is neither a processing success nor an error
# (the hop is its own event, counted via record_document_escalation).
if escalate_error_cls is not None and isinstance(e, escalate_error_cls):
raise
# Single processing-error call site: catches exhausted-retry
# re-raises, delete failures, and setup errors (get_qdrant_client /
# get_settings) — each counted exactly once. A failed delete is not
@@ -571,11 +686,20 @@ async def process_document(
async def _index_document(
doc_task: DocumentTask, nc_client: NextcloudClient, qdrant_client
doc_task: DocumentTask,
nc_client: NextcloudClient,
qdrant_client,
*,
tier: str | None = None,
) -> bool | None:
"""
Index a single document (called by process_document with retry).
``tier`` selects the external per-tier PDF path (Deck #323): when set and the
file is a PDF, exactly that tier is parsed and a low-quality result raises
``EscalateError`` to hand the document to the next tier's queue. ``None``
(default) runs the inline tiered pipeline (``registry.process``).
Returns ``False`` when a permanent parse failure means nothing was indexed
(the caller must then skip the success metrics); ``None`` otherwise.
@@ -800,22 +924,40 @@ async def _index_document(
"vector_sync.file_size": len(content_bytes),
},
):
# The registry runs the tiered PDF pipeline (tier-0 classify ->
# tier-1 fast -> OCR escalation) and records classification metrics.
# Imported lazily so module import doesn't pull in the document stack
# (document_processors -> _isolation, Unix-only ``resource``; see #877).
# The registry runs the tiered PDF pipeline and records
# classification metrics. Imported lazily so module import doesn't
# pull in the document stack (document_processors -> _isolation,
# Unix-only ``resource``; see #877).
from nextcloud_mcp_server.document_processors import ( # noqa: PLC0415
get_registry,
)
from nextcloud_mcp_server.document_processors.escalation import ( # noqa: PLC0415
EscalateError,
)
registry = get_registry()
try:
result = await registry.process(
content=content_bytes,
content_type=content_type,
filename=file_path,
)
# External per-tier path (Deck #323): run only this worker's tier
# for PDFs and let a low-quality parse raise EscalateError (a
# queue-hop to the next tier). Everything else -- non-PDF files,
# and the in-process/memory pool (tier is None) -- runs the inline
# tiered pipeline (fast -> OCR escalation in one call).
if tier is not None and _is_pdf(content_type):
result = await _parse_pdf_tier(
registry,
content_bytes,
content_type,
file_path,
tier,
settings,
)
else:
result = await registry.process(
content=content_bytes,
content_type=content_type,
filename=file_path,
)
# A permanent parse failure (e.g. an isolated-worker OOM/timeout
# on a pathological PDF) returns success=False rather than
@@ -881,6 +1023,11 @@ async def _index_document(
)
else:
logger.debug("No page_boundaries in metadata for %s", file_path)
except EscalateError:
# Control-flow signal (per-tier path): re-raise untouched so the
# queue hops the job to the next tier. NOT a "failed to process"
# error -- don't log it as one.
raise
except Exception as e:
logger.error("Failed to process file %s: %s", file_path, e)
raise
@@ -1037,6 +1184,14 @@ async def _index_document(
and not isinstance(raw_page_count, bool)
else None
),
# Tier that produced the parsed pages (registry stamps it on the
# result metadata); text doc types stay "fast". Narrow defensively
# to str|None — file_metadata is loosely typed (Any values).
pipeline_tier=(
pt
if isinstance(pt := file_metadata.get("pipeline_tier"), str)
else None
),
)
async def generate_sparse_embeddings():