Chris CoutinhoandClaude Opus 4.8
258ee96f4c
From card 309 (OHR-Bench smoke-test triage): during a backend-pod rollover the
embedding endpoint was briefly unreachable, and openai.APIConnectionError /
ConnectError propagated unretried (the provider only retried 429). Documents
exhausted the 3 in-process retries and were dropped for that scan cycle.
Broaden the provider-level retry to the transient set -- APIConnectionError,
APITimeoutError, 429, and 5xx -- on the existing exponential backoff (2s->60s,
5 attempts), so a few seconds of retry rides through the rollover. Permanent
4xx (auth, bad request) still re-raise immediately. Generalize the shared
_retry helper (retry_on_rate_limit -> retry_on_transient, predicate renamed to
should_retry, accurate log label) with a back-compat alias; Mistral gets 429+5xx
for parity. The production gateway path inherits this via GatewayProvider, which
delegates to the decorated OpenAIProvider methods.
Add astrolabe_vector_ingest_dropped_total{reason}, incremented when a document
exhausts retries, classified (connection|timeout|rate_limit|server|qdrant|other)
by _drop_reason so the embed-drop rate is alertable per cause. Dropped docs are
NOT marked failed, so the next full scan re-picks them (re-queue via scan loop).
Refs: Deck board 12 card 309 (AC #1 no permanently-dropped docs; embed-drop
metric for AC #5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
93 lines
3.4 KiB
Python
93 lines
3.4 KiB
Python
"""Shared transient-error retry helper for provider modules.
|
|
|
|
OpenAI and Mistral retry transient failures (429 rate limits, plus connection
|
|
drops / timeouts / 5xx for the embedding path) on the same exponential-backoff
|
|
curve; extracting the loop here keeps the two provider modules thin and lets
|
|
future providers (Bedrock throttling, etc.) reuse the same primitive. The
|
|
``should_retry`` predicate decides which caught exceptions are transient.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from collections.abc import Awaitable, Callable
|
|
from functools import wraps
|
|
from typing import Any, TypeVar
|
|
|
|
import anyio
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
MAX_RETRIES = 5
|
|
INITIAL_RETRY_DELAY = 2.0
|
|
MAX_RETRY_DELAY = 60.0
|
|
|
|
T = TypeVar("T")
|
|
|
|
|
|
def retry_on_transient(
|
|
exception_type: type[BaseException] | tuple[type[BaseException], ...],
|
|
should_retry: Callable[[BaseException], bool] = lambda _exc: True,
|
|
*,
|
|
provider_name: str = "provider",
|
|
label: str = "rate limit",
|
|
) -> Callable[[Callable[..., Awaitable[T]]], Callable[..., Awaitable[T]]]:
|
|
"""Build a decorator that retries transient exceptions with backoff.
|
|
|
|
Args:
|
|
exception_type: Catch this exception class (or tuple of classes), e.g.
|
|
``openai.APIError`` or ``mistralai.client.errors.SDKError``.
|
|
should_retry: Predicate that decides whether a caught exception is
|
|
transient (and so retryable) vs. a permanent error of the same
|
|
class. Defaults to "always True" — appropriate when
|
|
``exception_type`` is already transient-specific (e.g. a 429 class).
|
|
provider_name: Used in log messages so operators can tell which
|
|
provider exhausted retries.
|
|
label: Short noun for the log message ("rate limit", "transient error")
|
|
so the line accurately names what was retried.
|
|
"""
|
|
|
|
def decorator(func: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]:
|
|
@wraps(func)
|
|
async def wrapper(*args: Any, **kwargs: Any) -> T:
|
|
retry_delay = INITIAL_RETRY_DELAY
|
|
last_error: BaseException | None = None
|
|
|
|
for attempt in range(1, MAX_RETRIES + 1):
|
|
try:
|
|
return await func(*args, **kwargs)
|
|
except exception_type as e:
|
|
if not should_retry(e):
|
|
raise
|
|
last_error = e
|
|
if attempt < MAX_RETRIES:
|
|
logger.warning(
|
|
"%s %s (attempt %d/%d): %r; retrying in %.1fs...",
|
|
provider_name,
|
|
label,
|
|
attempt,
|
|
MAX_RETRIES,
|
|
e,
|
|
retry_delay,
|
|
)
|
|
await anyio.sleep(retry_delay)
|
|
retry_delay = min(retry_delay * 2, MAX_RETRY_DELAY)
|
|
|
|
logger.error(
|
|
"%s %s not resolved after %d attempts",
|
|
provider_name,
|
|
label,
|
|
MAX_RETRIES,
|
|
)
|
|
if last_error is None: # pragma: no cover — loop above always sets this
|
|
raise RuntimeError("retry loop exited without capturing an error")
|
|
raise last_error
|
|
|
|
return wrapper
|
|
|
|
return decorator
|
|
|
|
|
|
# Back-compat alias: the helper was originally rate-limit-specific.
|
|
retry_on_rate_limit = retry_on_transient
|