fix(vector): retry transient embed errors so a pod rollover drops 0 docs

From card 309 (OHR-Bench smoke-test triage): during a backend-pod rollover the
embedding endpoint was briefly unreachable, and openai.APIConnectionError /
ConnectError propagated unretried (the provider only retried 429). Documents
exhausted the 3 in-process retries and were dropped for that scan cycle.

Broaden the provider-level retry to the transient set -- APIConnectionError,
APITimeoutError, 429, and 5xx -- on the existing exponential backoff (2s->60s,
5 attempts), so a few seconds of retry rides through the rollover. Permanent
4xx (auth, bad request) still re-raise immediately. Generalize the shared
_retry helper (retry_on_rate_limit -> retry_on_transient, predicate renamed to
should_retry, accurate log label) with a back-compat alias; Mistral gets 429+5xx
for parity. The production gateway path inherits this via GatewayProvider, which
delegates to the decorated OpenAIProvider methods.

Add astrolabe_vector_ingest_dropped_total{reason}, incremented when a document
exhausts retries, classified (connection|timeout|rate_limit|server|qdrant|other)
by _drop_reason so the embed-drop rate is alertable per cause. Dropped docs are
NOT marked failed, so the next full scan re-picks them (re-queue via scan loop).

Refs: Deck board 12 card 309 (AC #1 no permanently-dropped docs; embed-drop
metric for AC #5).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Coutinho
2026-06-11 05:22:31 +02:00
co-authored by Claude Opus 4.8
parent 457c115ef4
commit 258ee96f4c
9 changed files with 344 additions and 53 deletions
+9 -6
View File
@@ -9,7 +9,7 @@ from nextcloud_mcp_server.providers.mistral import (
BATCH_SIZE,
MISTRAL_EMBEDDING_DIMENSIONS,
MistralProvider,
_is_rate_limit,
_is_transient,
)
@@ -318,14 +318,17 @@ async def test_mistral_embed_with_usage_single(mock_mistral_client):
@pytest.mark.unit
def test_mistral_is_rate_limit_predicate():
"""_is_rate_limit returns True only for SDKErrors with status_code == 429."""
def test_mistral_is_transient_predicate():
"""_is_transient retries 429 (rate limit) and 5xx (server/transient) SDKErrors."""
err_429 = MagicMock(spec=SDKError)
err_429.status_code = 429
err_500 = MagicMock(spec=SDKError)
err_500.status_code = 500
err_400 = MagicMock(spec=SDKError)
err_400.status_code = 400
assert _is_rate_limit(err_429) is True
assert _is_rate_limit(err_500) is False
assert _is_transient(err_429) is True
assert _is_transient(err_500) is True # broadened to 5xx (card 309)
assert _is_transient(err_400) is False # permanent client error
# ValueError has no status_code attr → getattr returns None → False.
assert _is_rate_limit(ValueError()) is False
assert _is_transient(ValueError()) is False