feat(ingest): per-tier escalation via procrastinate queue-hop
Split external (procrastinate) document processing into per-tier queues so a
document is attempted at most once per tier and requeued to the next tier's
queue on a low-quality parse, using procrastinate's native retry.
- escalation.py: TIER_LADDER (fast->structured->ocr) + EscalateError signal
- registry: process_tier (one tier) + evaluate_escalation post-parse gate
(reuses classify_from_text) + next_available_tier; shared _classify_result
and _oversize_result with the inline pipeline
- processor: process_document(tier=...) runs one tier and raises EscalateError
before embed (junk text never indexed); inline memory path unchanged
- queue/procrastinate: ingest-fast|structured|ocr queues; TieredEscalationStrategy
(queue-hop on EscalateError, bounded same-tier transient retry); queue-aware
task; producer defers to ingest-fast; per-queue counts + all-queue reclaim
- cli: worker --tier {fast,structured,ocr}
- billing: pages_ocr usage event + pipeline_tier metadata (paid OCR billed apart)
- observability: astrolabe_ingest_queue_depth{queue,status} gauge + per-queue
counts in nc_get_vector_sync_status / management status endpoint
- config: INGEST_ESCALATION_ENABLED (default true), INGEST_TRANSIENT_MAX_ATTEMPTS
INGEST_ESCALATION_ENABLED=false and INGEST_QUEUE=memory preserve prior behaviour.
Deck #323.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
6fab0e2ae3
commit
9676bb3106
@@ -176,3 +176,53 @@ async def test_store_failure_is_swallowed(monkeypatch):
|
||||
total_chars=9,
|
||||
page_count=2,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
async def test_ocr_tier_records_pages_ocr(store_spy):
|
||||
"""OCR-tier pages are metered as a separate pages_ocr line (Deck #323)."""
|
||||
await processor.record_indexing_usage(
|
||||
enabled=True,
|
||||
provider="mistral",
|
||||
model="mistral-embed",
|
||||
doc_type="file",
|
||||
user_id="alice",
|
||||
chunk_count=20,
|
||||
token_count=900,
|
||||
total_chars=40000,
|
||||
page_count=8,
|
||||
pipeline_tier="ocr",
|
||||
)
|
||||
by_metric = {
|
||||
c.kwargs["metric"]: c.kwargs["value"]
|
||||
for c in store_spy.record_usage_event.await_args_list
|
||||
}
|
||||
# pages_ocr fires IN ADDITION to pages_embedded for OCR-tier pages.
|
||||
assert by_metric == {
|
||||
"tokens_embedded": 900,
|
||||
"pages_embedded": 8,
|
||||
"pages_ocr": 8,
|
||||
}
|
||||
# pipeline_tier is threaded into the billing metadata for CP attribution.
|
||||
for c in store_spy.record_usage_event.await_args_list:
|
||||
assert c.kwargs["metadata"]["pipeline_tier"] == "ocr"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
async def test_fast_tier_does_not_record_pages_ocr(store_spy):
|
||||
"""A CPU-cheap fast-tier parse must NOT incur the paid pages_ocr line."""
|
||||
await processor.record_indexing_usage(
|
||||
enabled=True,
|
||||
provider="mistral",
|
||||
model="mistral-embed",
|
||||
doc_type="file",
|
||||
user_id="alice",
|
||||
chunk_count=10,
|
||||
token_count=500,
|
||||
total_chars=20000,
|
||||
page_count=4,
|
||||
pipeline_tier="fast",
|
||||
)
|
||||
metrics = {c.kwargs["metric"] for c in store_spy.record_usage_event.await_args_list}
|
||||
assert "pages_ocr" not in metrics
|
||||
assert metrics == {"tokens_embedded", "pages_embedded"}
|
||||
|
||||
Reference in New Issue
Block a user