admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(ingest): per-tier escalation via procrastinate queue-hop

Browse Source 
Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.

- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
  (reuses classify_from_text) + next_available_tier; shared _classify_result
  and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
  before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
  (queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
  task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
  counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS

INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.

Deck #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-13 13:22:18 +02:00
co-authored by Claude Opus 4.8
parent 6fab0e2ae3
commit 9676bb3106
17 changed files with 1259 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nextcloud_mcp_server/vector/queue/procrastinate.py
+253 -29
View File
@@ -5,11 +5,14 @@ This replaces NATS JetStream and the old Postgres-queue stub. The MCP server now
owns *both* sides of ingest:
- **Producer** (API role / scanner) — :class:`ProcrastinateTaskProducer.send`
*defers* one ``ingest:process_document`` job per changed document into the
per-tenant Postgres (the same app DB; procrastinate manages its own tables).
- **Consumer** (worker role) — ``nextcloud-mcp-server worker`` runs
:func:`procrastinate.App.run_worker`, which drains the ``ingest`` queue and
invokes the existing :func:`process_document` pipeline.
*defers* one ``ingest:process_document`` job per changed document onto the
cheapest tier's queue (``ingest-fast``) in the per-tenant Postgres (the same
app DB; procrastinate manages its own tables).
- **Consumer** (worker role) — ``nextcloud-mcp-server worker [--tier T]`` runs
:func:`procrastinate.App.run_worker`, which drains its tier's queue and invokes
the existing :func:`process_document` pipeline. A parse too poor to index hops
the job to the next tier's queue (see :class:`TieredEscalationStrategy`), so
cheap CPU parsing and paid OCR run on independently-scaled fleets (Deck #323).
Design notes:
@@ -35,9 +38,17 @@ from datetime import datetime, timezone
from types import TracebackType
from typing import TYPE_CHECKING
from procrastinate import App, Blueprint, JobContext, PsycopgConnector, RetryStrategy
from procrastinate import (
App,
BaseRetryStrategy,
Blueprint,
JobContext,
PsycopgConnector,
RetryDecision,
)
from procrastinate.connector import BaseConnector
from procrastinate.exceptions import AlreadyEnqueued
from procrastinate.jobs import Job
from ...config import get_procrastinate_conninfo, get_settings
from ..scanner import DocumentTask
@@ -47,14 +58,53 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
# Single queue for document ingest. KEDA scales the worker Deployment on the
# depth of this queue (``SELECT count(*) FROM procrastinate_jobs WHERE
# queue_name='ingest' AND status='todo'``).
INGEST_QUEUE_NAME = "ingest"
# One queue per extraction tier (Deck #323), aligned cheapest-first with
# document_processors.escalation.TIER_LADDER. Each queue is drained by its own
# worker Deployment + KEDA ScaledObject (``SELECT count(*) FROM
# procrastinate_jobs WHERE queue_name=<queue> AND status='todo'``), so a
# CPU-bound ``fast`` fleet, an in-cluster ``structured`` fleet, and a paid
# network-bound ``ocr`` fleet scale (and fail) independently.
INGEST_QUEUE_FAST = "ingest-fast"
INGEST_QUEUE_STRUCTURED = "ingest-structured"
INGEST_QUEUE_OCR = "ingest-ocr"
# tier -> queue. The producer always defers onto the cheapest tier's queue; a
# low-quality parse hops the job up the ladder via the retry strategy below.
TIER_QUEUES: dict[str, str] = {
"fast": INGEST_QUEUE_FAST,
"structured": INGEST_QUEUE_STRUCTURED,
"ocr": INGEST_QUEUE_OCR,
}
_QUEUE_TIERS: dict[str, str] = {queue: tier for tier, queue in TIER_QUEUES.items()}
ALL_INGEST_QUEUES: tuple[str, ...] = tuple(TIER_QUEUES.values())
# New jobs start here; ``ocr`` is reached only by escalation.
DEFAULT_INGEST_QUEUE = INGEST_QUEUE_FAST
# Legacy single-queue name (pre-#323). A rolling upgrade may still have jobs
# parked on it; a worker can be told to drain it alongside the tier queues, and
# the job-count / reclaim helpers include it so nothing is stranded.
LEGACY_INGEST_QUEUE = "ingest"
# Back-compat alias for callers that imported the old single-queue constant.
INGEST_QUEUE_NAME = DEFAULT_INGEST_QUEUE
# Queues the job-count + reclaim helpers sweep (tier queues + the legacy one).
_MANAGED_QUEUES: tuple[str, ...] = (*ALL_INGEST_QUEUES, LEGACY_INGEST_QUEUE)
# Blueprint namespace → registered task names are prefixed ``ingest:``.
_NAMESPACE = "ingest"
INGEST_TASK_NAME = f"{_NAMESPACE}:process_document"
def tier_for_queue(queue: str | None) -> str:
"""Tier a worker on ``queue`` should run. Unknown/legacy -> ``fast``.
The queue-aware task uses this to pick which single tier to parse with: the
job's current queue *is* its tier. A job on the legacy ``ingest`` queue (or
any unrecognised queue) defaults to the cheapest tier.
"""
return _QUEUE_TIERS.get(queue or "", "fast")
# A crashed worker leaves its job in ``doing``; reclaim it once its (per-worker)
# heartbeat is this many seconds stale. The default is sized well above the
# longest expected ``process_document`` (PDF render + embedding) so a slow-but-
@@ -69,6 +119,7 @@ INGEST_TASK_NAME = f"{_NAMESPACE}:process_document"
# Blueprint cannot be added to more than one App — which the tests (in-memory +
# real Postgres) and any re-init path require.
async def process_document_task(
context: JobContext,
*,
user_id: str,
doc_id: str,
@@ -80,12 +131,26 @@ async def process_document_task(
etag: str | None = None,
owner_id: str | None = None,
) -> None:
"""Worker entry: rebuild the DocumentTask, resolve creds, run the pipeline."""
"""Worker entry: rebuild the DocumentTask, resolve creds, run the pipeline.
Queue-aware (Deck #323): the tier this worker runs is the tier of the job's
current queue. A low-quality parse raises ``EscalateError``, which the
:class:`TieredEscalationStrategy` turns into a queue-hop to the next tier.
When per-tier escalation is disabled (``INGEST_ESCALATION_ENABLED=false``),
``tier`` stays ``None`` and the inline pipeline runs (fast -> OCR in one
call), reproducing the pre-#323 single-queue behaviour.
"""
# Local imports avoid a heavy import chain at blueprint-definition time
# (this module is also imported by the API pod just to defer jobs).
from ..oauth_sync import NotProvisionedError # noqa: PLC0415
from ..processor import process_document # noqa: PLC0415
tier = (
tier_for_queue(context.job.queue)
if get_settings().ingest_escalation_enabled
else None
)
task = DocumentTask(
user_id=user_id,
doc_id=doc_id,
@@ -111,7 +176,7 @@ async def process_document_task(
try:
# Durable retry is procrastinate's job; disable the in-process loop.
await process_document(task, nc_client, max_retries=1)
await process_document(task, nc_client, max_retries=1, tier=tier)
finally:
await nc_client.close()
@@ -127,9 +192,11 @@ async def reclaim_stalled_ingest_jobs(context: JobContext, timestamp: int) -> No
retry_at = datetime.now(tz=timezone.utc)
stalled_after = get_settings().ingest_stalled_job_seconds
reclaimed = 0
for job in await manager.get_stalled_jobs(
queue=INGEST_QUEUE_NAME, seconds_since_heartbeat=stalled_after
):
# queue=None sweeps every queue, so an orphaned job on any tier queue is
# reclaimed regardless of which tier's worker happens to run this periodic.
# retry_job_by_id_async keeps the job on its own queue, so a stalled ``ocr``
# job re-runs on ``ingest-ocr`` (the ocr fleet), not the reclaiming worker's.
for job in await manager.get_stalled_jobs(seconds_since_heartbeat=stalled_after):
if job.id is None:
continue
await manager.retry_job_by_id_async(job_id=job.id, retry_at=retry_at)
@@ -163,6 +230,121 @@ async def _resolve_client(user_id: str) -> NextcloudClient:
return await get_user_client_basic_auth(user_id, host)
def _first_leaf(exc: BaseException) -> BaseException:
"""Descend nested ExceptionGroups to the first concrete leaf exception.
An anyio task group can wrap the real cause (and nest groups); the retry
strategy classifies on the leaf, mirroring ``processor._drop_reason``.
"""
while isinstance(exc, BaseExceptionGroup) and exc.exceptions:
exc = exc.exceptions[0]
return exc
def _is_transient_infra_error(exc: BaseException) -> bool:
"""Whether ``exc`` is a transient infra blip worth a SAME-tier retry.
Mirrors the retryable subset of ``processor._drop_reason``: doc-fetch /
embed / Qdrant timeouts, connection drops, rate limits, and 5xx. A parse
that is merely *poor* never reaches here -- that path raises
``EscalateError`` (handled separately) -- so this is purely about
infrastructure that should recover on its own. Imports are lazy: this only
runs in the worker, and the module is also imported by the API pod to defer.
"""
import httpx # noqa: PLC0415
if isinstance(exc, (httpx.TimeoutException, httpx.ConnectError)):
return True
try:
import openai # noqa: PLC0415
if isinstance(
exc,
(
openai.APITimeoutError,
openai.APIConnectionError,
openai.RateLimitError,
),
):
return True
if isinstance(exc, openai.APIStatusError):
return exc.status_code >= 500
except ImportError: # pragma: no cover -- openai is a hard dependency
pass
if type(exc).__module__.startswith("qdrant_client"):
return True
return False
class TieredEscalationStrategy(BaseRetryStrategy):
"""Native procrastinate retry that escalates across tier queues (Deck #323).
Three outcomes, decided from the raised exception:
- ``EscalateError`` -> ``RetryDecision(queue=<next tier's queue>)``: the SAME
job hops to the next fleet's queue and is parsed once by that tier. This is
how a document is "requeued on a failed parse" -- once per tier, with no
same-tier parse retry.
- a whitelisted transient infra error (doc fetch / embed / Qdrant blip) ->
same-queue exponential backoff, while under ``max_transient_attempts``.
- anything else, the transient cap is reached, or the target tier is unknown
-> ``None`` (no retry); the placeholder was already marked failed by the
pipeline, and the next scan re-picks the document.
Per-tier attempt accounting is intentionally approximate: a queue-hop can't
reset ``job.attempts`` (procrastinate has no per-tier counter), so parse
escalations *do* advance the same counter the transient cap reads. Because a
parse escalation hops (it never retries in place) the "once per parse per
tier" guarantee is structural; the cap is just a generous global ceiling on
transient churn across the whole lineage, not an exact per-tier count.
"""
def __init__(self, *, max_transient_attempts: int) -> None:
self._max_transient_attempts = max_transient_attempts
def get_retry_decision(
self, *, exception: BaseException, job: Job
) -> RetryDecision | None:
# Lazy import: EscalateError lives in the document stack, which the API
# pod (it also builds this App to defer) must not load. get_retry_decision
# runs only in the worker, where the stack is already imported.
from ...document_processors.escalation import EscalateError # noqa: PLC0415
exc = _first_leaf(exception)
if isinstance(exc, EscalateError):
queue = TIER_QUEUES.get(exc.to_tier)
if queue is None:
# Unknown target tier: don't strand the job on a queue no worker
# drains -- stop and let the placeholder/next scan handle it.
logger.error(
"ingest.escalate_unknown_tier from=%s to=%s",
exc.from_tier,
exc.to_tier,
)
return None
logger.info(
"ingest.escalate from=%s to=%s reason=%s queue=%s",
exc.from_tier,
exc.to_tier,
exc.reason,
queue,
)
# Immediate hop -- the next tier's fleet should pick it up at once.
return RetryDecision(queue=queue, retry_in={"seconds": 0})
if (
_is_transient_infra_error(exc)
and job.attempts < self._max_transient_attempts
):
# 4, 8, 16, ... seconds, capped at 5 min. attempts is >=1 here (the
# failing attempt is counted), so attempts-1 makes the first wait 4s.
wait = min(4 * (2 ** max(0, job.attempts - 1)), 300)
return RetryDecision(retry_in={"seconds": wait})
return None
def _build_ingest_blueprint() -> Blueprint:
"""Create a fresh Blueprint with the ingest tasks registered.
@@ -172,13 +354,22 @@ def _build_ingest_blueprint() -> Blueprint:
bp = Blueprint()
# Durable retry owned by the queue (survives worker crashes); the in-process
# retry loop in process_document is disabled on this path via max_retries=1.
bp.task(
# The task's default queue is the cheapest tier; the producer defers there
# explicitly and the strategy hops a job up the ladder on a poor parse.
bp.task( # type: ignore[no-matching-overload]
name="process_document",
queue=INGEST_QUEUE_NAME,
retry=RetryStrategy(max_attempts=5, exponential_wait=4),
queue=DEFAULT_INGEST_QUEUE,
pass_context=True,
# procrastinate's RetryValue type only admits RetryStrategy, but a custom
# BaseRetryStrategy subclass is the documented extension point (and is
# accepted at runtime by get_retry_strategy). The annotation is just too
# narrow, hence the ignore.
retry=TieredEscalationStrategy(
max_transient_attempts=get_settings().ingest_transient_max_attempts
),
)(process_document_task)
reclaim = bp.task(
name="reclaim_stalled_jobs", queue=INGEST_QUEUE_NAME, pass_context=True
name="reclaim_stalled_jobs", queue=DEFAULT_INGEST_QUEUE, pass_context=True
)(reclaim_stalled_ingest_jobs)
bp.periodic(cron="*/5 * * * *", periodic_id="reclaim_stalled_ingest")(reclaim)
return bp
@@ -276,20 +467,43 @@ async def apply_ingest_queue_schema(
_JOB_STATUSES = ("todo", "doing", "succeeded", "failed", "cancelled", "aborted")
async def get_ingest_job_counts(app: App | None = None) -> dict[str, int]:
"""Return ingest job counts by status (``todo``/``doing``/``failed``/…).
async def get_ingest_job_counts_by_queue(
app: App | None = None,
) -> dict[str, dict[str, int]]:
"""Per-queue ingest job counts by status (Deck #323).
Reads procrastinate's per-queue stats via the manager API (not hand-written
SQL) so a future schema bump doesn't silently break the status surface. The
manager flattens its per-status ``stats`` into top-level row keys, so we read
the known status keys directly. Assumes the app's connector is already open.
Returns ``{queue_name: {status: count}}`` for the managed ingest queues (the
per-tier queues + the legacy single queue) that have rows. Reads
procrastinate's per-queue stats via the manager API (not hand-written SQL) so
a future schema bump doesn't silently break the status surface. Assumes the
app's connector is already open. Feeds the per-tier status surface + the
``astrolabe_ingest_queue_depth`` gauge.
"""
app = app or get_procrastinate_app()
counts: dict[str, int] = {}
for row in await app.job_manager.list_queues_async(queue=INGEST_QUEUE_NAME):
by_queue: dict[str, dict[str, int]] = {}
for row in await app.job_manager.list_queues_async():
name = row.get("name")
if name not in _MANAGED_QUEUES:
continue
per = by_queue.setdefault(name, {})
for status in _JOB_STATUSES:
if status in row:
counts[status] = counts.get(status, 0) + int(row[status])
per[status] = per.get(status, 0) + int(row[status])
return by_queue
async def get_ingest_job_counts(app: App | None = None) -> dict[str, int]:
"""Aggregate ingest job counts by status across all managed queues.
Fleet-wide totals summed over the per-tier queues + the legacy queue, so
``pending = todo + doing`` reflects all outstanding ingest work regardless of
which tier a document currently sits on. Per-queue breakdown:
:func:`get_ingest_job_counts_by_queue`.
"""
counts: dict[str, int] = {}
for per in (await get_ingest_job_counts_by_queue(app)).values():
for status, value in per.items():
counts[status] = counts.get(status, 0) + value
return counts
@@ -336,7 +550,13 @@ class ProcrastinateTaskProducer:
async def send(self, task: DocumentTask, /) -> None:
key = _doc_queueing_lock(task)
deferrer = self._app.configure_task(INGEST_TASK_NAME, queueing_lock=key)
# Always defer onto the cheapest tier's queue; the escalation strategy
# hops the job up the ladder on a poor parse. queueing_lock is a global
# partial-unique on status='todo', so a doc mid-escalation on a higher
# tier still dedupes a fresh enqueue here -- no double-processing.
deferrer = self._app.configure_task(
INGEST_TASK_NAME, queue=DEFAULT_INGEST_QUEUE, queueing_lock=key
)
try:
await deferrer.defer_async(**asdict(task))
except AlreadyEnqueued:
@@ -358,6 +578,10 @@ class ProcrastinateTaskProducer:
"""Ingest job counts by status (for the vector-sync status surface)."""
return await get_ingest_job_counts(self._app)
async def job_counts_by_queue(self) -> dict[str, dict[str, int]]:
"""Per-tier-queue ingest job counts by status (Deck #323)."""
return await get_ingest_job_counts_by_queue(self._app)
def clone(self) -> ProcrastinateTaskProducer:
return self