feat(ingest): per-tier escalation via procrastinate queue-hop
Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.
- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
(reuses classify_from_text) + next_available_tier; shared _classify_result
and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
(queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS
INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
6fab0e2ae3
commit
9676bb3106
@@ -333,8 +333,18 @@ def _init_worker_observability(settings: Settings) -> None:
|
||||
default=None,
|
||||
help="Max concurrent jobs. Defaults to VECTOR_SYNC_PROCESSOR_WORKERS.",
|
||||
)
|
||||
def worker(concurrency: int | None):
|
||||
"""Run the ingest worker (Deck #183).
|
||||
@click.option(
|
||||
"--tier",
|
||||
type=click.Choice(["fast", "structured", "ocr"]),
|
||||
default=None,
|
||||
help=(
|
||||
"Run only this extraction tier's queue (Deck #323). Omit to drain ALL "
|
||||
"tier queues in one process (single-Deployment / dev); set it to run one "
|
||||
"tier per Deployment so the fleets scale independently."
|
||||
),
|
||||
)
|
||||
def worker(concurrency: int | None, tier: str | None):
|
||||
"""Run the ingest worker (Deck #183, per-tier fleets #323).
|
||||
|
||||
\b
|
||||
Drains the per-tenant Postgres ingest queue (procrastinate): for each
|
||||
@@ -342,6 +352,13 @@ def worker(concurrency: int | None):
|
||||
embeds, and upserts into Qdrant. This is the scale-to-zero ``worker`` role of
|
||||
the api/worker split; run it as a separate Deployment from the API pod.
|
||||
|
||||
\b
|
||||
With --tier the worker drains only that tier's queue (``ingest-<tier>``), so
|
||||
a CPU-bound ``fast`` fleet, an in-cluster ``structured`` fleet, and a paid
|
||||
``ocr`` fleet scale independently. Without it, all tier queues are drained in
|
||||
one process (handy for dev / a single Deployment). A low-quality parse hops
|
||||
the job to the next tier's queue automatically (see TieredEscalationStrategy).
|
||||
|
||||
\b
|
||||
Requires INGEST_QUEUE=postgres (a PostgreSQL DATABASE_URL); procrastinate is
|
||||
Postgres-only.
|
||||
@@ -349,7 +366,7 @@ def worker(concurrency: int | None):
|
||||
\b
|
||||
Example:
|
||||
$ export DATABASE_URL=postgresql+asyncpg://mcp:mcp@db/mcp
|
||||
$ nextcloud-mcp-server worker -c 4
|
||||
$ nextcloud-mcp-server worker -c 4 --tier fast
|
||||
"""
|
||||
import anyio # noqa: PLC0415
|
||||
|
||||
@@ -366,11 +383,21 @@ def worker(concurrency: int | None):
|
||||
_init_worker_observability(settings)
|
||||
|
||||
from nextcloud_mcp_server.vector.queue.procrastinate import ( # noqa: PLC0415
|
||||
INGEST_QUEUE_NAME,
|
||||
ALL_INGEST_QUEUES,
|
||||
LEGACY_INGEST_QUEUE,
|
||||
TIER_QUEUES,
|
||||
apply_ingest_queue_schema,
|
||||
get_procrastinate_app,
|
||||
)
|
||||
|
||||
# Which queues this process drains. A single tier -> just its queue; no tier
|
||||
# -> every tier queue PLUS the legacy single queue, so a rolling upgrade
|
||||
# never strands jobs deferred under the pre-#323 name.
|
||||
if tier is not None:
|
||||
queues = [TIER_QUEUES[tier]]
|
||||
else:
|
||||
queues = [*ALL_INGEST_QUEUES, LEGACY_INGEST_QUEUE]
|
||||
|
||||
# This is the consumer side of the distributed (postgres) ingest backend.
|
||||
# Unlike the in-process anyio pool, the worker talks to procrastinate's App
|
||||
# directly (run_worker_async), so it does NOT go through IngestTransport —
|
||||
@@ -397,13 +424,15 @@ def worker(concurrency: int | None):
|
||||
# Structured log (not click.echo) so it lands in the JSON / OTel
|
||||
# pipeline like every other startup message.
|
||||
logger.info(
|
||||
"Ingest worker started: queue=%s concurrency=%s delete_succeeded=%s",
|
||||
INGEST_QUEUE_NAME,
|
||||
"Ingest worker started: tier=%s queues=%s concurrency=%s "
|
||||
"delete_succeeded=%s",
|
||||
tier or "all",
|
||||
queues,
|
||||
workers,
|
||||
settings.ingest_delete_succeeded_jobs,
|
||||
)
|
||||
await app.run_worker_async(
|
||||
queues=[INGEST_QUEUE_NAME],
|
||||
queues=queues,
|
||||
concurrency=workers,
|
||||
install_signal_handlers=True,
|
||||
# Drop succeeded jobs (default) so the queue table stays lean and
|
||||
|
||||
Reference in New Issue
Block a user