fix(vector): retry transient embed errors so a pod rollover drops 0 docs
From card 309 (OHR-Bench smoke-test triage): during a backend-pod rollover the
embedding endpoint was briefly unreachable, and openai.APIConnectionError /
ConnectError propagated unretried (the provider only retried 429). Documents
exhausted the 3 in-process retries and were dropped for that scan cycle.
Broaden the provider-level retry to the transient set -- APIConnectionError,
APITimeoutError, 429, and 5xx -- on the existing exponential backoff (2s->60s,
5 attempts), so a few seconds of retry rides through the rollover. Permanent
4xx (auth, bad request) still re-raise immediately. Generalize the shared
_retry helper (retry_on_rate_limit -> retry_on_transient, predicate renamed to
should_retry, accurate log label) with a back-compat alias; Mistral gets 429+5xx
for parity. The production gateway path inherits this via GatewayProvider, which
delegates to the decorated OpenAIProvider methods.
Add astrolabe_vector_ingest_dropped_total{reason}, incremented when a document
exhausts retries, classified (connection|timeout|rate_limit|server|qdrant|other)
by _drop_reason so the embed-drop rate is alertable per cause. Dropped docs are
NOT marked failed, so the next full scan re-picks them (re-queue via scan loop).
Refs: Deck board 12 card 309 (AC #1 no permanently-dropped docs; embed-drop
metric for AC #5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
457c115ef4
commit
258ee96f4c
@@ -9,7 +9,7 @@ from nextcloud_mcp_server.providers.mistral import (
|
||||
BATCH_SIZE,
|
||||
MISTRAL_EMBEDDING_DIMENSIONS,
|
||||
MistralProvider,
|
||||
_is_rate_limit,
|
||||
_is_transient,
|
||||
)
|
||||
|
||||
|
||||
@@ -318,14 +318,17 @@ async def test_mistral_embed_with_usage_single(mock_mistral_client):
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_mistral_is_rate_limit_predicate():
|
||||
"""_is_rate_limit returns True only for SDKErrors with status_code == 429."""
|
||||
def test_mistral_is_transient_predicate():
|
||||
"""_is_transient retries 429 (rate limit) and 5xx (server/transient) SDKErrors."""
|
||||
err_429 = MagicMock(spec=SDKError)
|
||||
err_429.status_code = 429
|
||||
err_500 = MagicMock(spec=SDKError)
|
||||
err_500.status_code = 500
|
||||
err_400 = MagicMock(spec=SDKError)
|
||||
err_400.status_code = 400
|
||||
|
||||
assert _is_rate_limit(err_429) is True
|
||||
assert _is_rate_limit(err_500) is False
|
||||
assert _is_transient(err_429) is True
|
||||
assert _is_transient(err_500) is True # broadened to 5xx (card 309)
|
||||
assert _is_transient(err_400) is False # permanent client error
|
||||
# ValueError has no status_code attr → getattr returns None → False.
|
||||
assert _is_rate_limit(ValueError()) is False
|
||||
assert _is_transient(ValueError()) is False
|
||||
|
||||
Reference in New Issue
Block a user