feat(ingest): per-tier escalation via procrastinate queue-hop

Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.

- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
  (reuses classify_from_text) + next_available_tier; shared _classify_result
  and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
  before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
  (queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
  task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
  counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS

INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.

Deck #323.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-13 13:22:18 +02:00
co-authored by Claude Opus 4.8
parent 6fab0e2ae3
commit 9676bb3106
17 changed files with 1259 additions and 129 deletions
+50
View File
@@ -176,3 +176,53 @@ async def test_store_failure_is_swallowed(monkeypatch):
total_chars=9,
page_count=2,
)
@pytest.mark.unit
async def test_ocr_tier_records_pages_ocr(store_spy):
"""OCR-tier pages are metered as a separate pages_ocr line (Deck #323)."""
await processor.record_indexing_usage(
enabled=True,
provider="mistral",
model="mistral-embed",
doc_type="file",
user_id="alice",
chunk_count=20,
token_count=900,
total_chars=40000,
page_count=8,
pipeline_tier="ocr",
)
by_metric = {
c.kwargs["metric"]: c.kwargs["value"]
for c in store_spy.record_usage_event.await_args_list
}
# pages_ocr fires IN ADDITION to pages_embedded for OCR-tier pages.
assert by_metric == {
"tokens_embedded": 900,
"pages_embedded": 8,
"pages_ocr": 8,
}
# pipeline_tier is threaded into the billing metadata for CP attribution.
for c in store_spy.record_usage_event.await_args_list:
assert c.kwargs["metadata"]["pipeline_tier"] == "ocr"
@pytest.mark.unit
async def test_fast_tier_does_not_record_pages_ocr(store_spy):
"""A CPU-cheap fast-tier parse must NOT incur the paid pages_ocr line."""
await processor.record_indexing_usage(
enabled=True,
provider="mistral",
model="mistral-embed",
doc_type="file",
user_id="alice",
chunk_count=10,
token_count=500,
total_chars=20000,
page_count=4,
pipeline_tier="fast",
)
metrics = {c.kwargs["metric"] for c in store_spy.record_usage_event.await_args_list}
assert "pages_ocr" not in metrics
assert metrics == {"tokens_embedded", "pages_embedded"}