feat(ingest): per-tier escalation via procrastinate queue-hop

Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.

- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
  (reuses classify_from_text) + next_available_tier; shared _classify_result
  and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
  before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
  (queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
  task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
  counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS

INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.

Deck #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-13 13:22:18 +02:00
co-authored by Claude Opus 4.8
parent 6fab0e2ae3
commit 9676bb3106
17 changed files with 1259 additions and 129 deletions
+36 -7
View File
@@ -333,8 +333,18 @@ def _init_worker_observability(settings: Settings) -> None:
default=None,
help="Max concurrent jobs. Defaults to VECTOR_SYNC_PROCESSOR_WORKERS.",
)
def worker(concurrency: int | None):
"""Run the ingest worker (Deck #183).
@click.option(
"--tier",
type=click.Choice(["fast", "structured", "ocr"]),
default=None,
help=(
"Run only this extraction tier's queue (Deck #323). Omit to drain ALL "
"tier queues in one process (single-Deployment / dev); set it to run one "
"tier per Deployment so the fleets scale independently."
),
)
def worker(concurrency: int | None, tier: str | None):
"""Run the ingest worker (Deck #183, per-tier fleets #323).
\b
Drains the per-tenant Postgres ingest queue (procrastinate): for each
@@ -342,6 +352,13 @@ def worker(concurrency: int | None):
embeds, and upserts into Qdrant. This is the scale-to-zero ``worker`` role of
the api/worker split; run it as a separate Deployment from the API pod.
\b
With --tier the worker drains only that tier's queue (``ingest-<tier>``), so
a CPU-bound ``fast`` fleet, an in-cluster ``structured`` fleet, and a paid
``ocr`` fleet scale independently. Without it, all tier queues are drained in
one process (handy for dev / a single Deployment). A low-quality parse hops
the job to the next tier's queue automatically (see TieredEscalationStrategy).
\b
Requires INGEST_QUEUE=postgres (a PostgreSQL DATABASE_URL); procrastinate is
Postgres-only.
@@ -349,7 +366,7 @@ def worker(concurrency: int | None):
\b
Example:
$ export DATABASE_URL=postgresql+asyncpg://mcp:mcp@db/mcp
$ nextcloud-mcp-server worker -c 4
$ nextcloud-mcp-server worker -c 4 --tier fast
"""
import anyio # noqa: PLC0415
@@ -366,11 +383,21 @@ def worker(concurrency: int | None):
_init_worker_observability(settings)
from nextcloud_mcp_server.vector.queue.procrastinate import ( # noqa: PLC0415
INGEST_QUEUE_NAME,
ALL_INGEST_QUEUES,
LEGACY_INGEST_QUEUE,
TIER_QUEUES,
apply_ingest_queue_schema,
get_procrastinate_app,
)
# Which queues this process drains. A single tier -> just its queue; no tier
# -> every tier queue PLUS the legacy single queue, so a rolling upgrade
# never strands jobs deferred under the pre-#323 name.
if tier is not None:
queues = [TIER_QUEUES[tier]]
else:
queues = [*ALL_INGEST_QUEUES, LEGACY_INGEST_QUEUE]
# This is the consumer side of the distributed (postgres) ingest backend.
# Unlike the in-process anyio pool, the worker talks to procrastinate's App
# directly (run_worker_async), so it does NOT go through IngestTransport —
@@ -397,13 +424,15 @@ def worker(concurrency: int | None):
# Structured log (not click.echo) so it lands in the JSON / OTel
# pipeline like every other startup message.
logger.info(
"Ingest worker started: queue=%s concurrency=%s delete_succeeded=%s",
INGEST_QUEUE_NAME,
"Ingest worker started: tier=%s queues=%s concurrency=%s "
"delete_succeeded=%s",
tier or "all",
queues,
workers,
settings.ingest_delete_succeeded_jobs,
)
await app.run_worker_async(
queues=[INGEST_QUEUE_NAME],
queues=queues,
concurrency=workers,
install_signal_handlers=True,
# Drop succeeded jobs (default) so the queue table stays lean and