fix(vector): nested-group drop classification + review/Sonar fixes (#893)
Round-1 review on PR #893: - _drop_reason now descends through nested ExceptionGroups to the first leaf (was single-level), so a doubly-wrapped cause isn't mislabelled "other"; added a nested-group test. Commented why both the httpx and openai isinstance branches exist (raw Nextcloud-API errors vs SDK-wrapped variants). - Documented that generate() intentionally shares the broadened transient retry (RAG sampling path), with the worst-case latency note. - Added a docstring note to process_document on how the provider-level retry (5x) layers over the outer loop (3x in-process / 1x procrastinate). - Added test_embed_batch_retries_on_connection_error for the batch path. - Renamed test_retry_reraises_non_rate_limit_immediately -> test_retry_reraises_when_predicate_returns_false (it tests the predicate, not a specific status). SonarCloud: - S5708 (BLOCKER) on the helper's dynamic `except exception_type`: the type is constrained to BaseException/tuple by the signature; suppressed with a justified NOSONAR. - S7503 (async without await) in the embed-retry test: use AsyncMock side_effect instead of a hand-rolled async function. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
258ee96f4c
commit
6c99906ed4
@@ -56,7 +56,10 @@ def retry_on_transient(
|
||||
for attempt in range(1, MAX_RETRIES + 1):
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except exception_type as e:
|
||||
# exception_type is constrained by the signature to a
|
||||
# BaseException subclass or a tuple of them; the dynamic catch is
|
||||
# the whole point of this reusable helper.
|
||||
except exception_type as e: # NOSONAR(S5708)
|
||||
if not should_retry(e):
|
||||
raise
|
||||
last_error = e
|
||||
|
||||
@@ -283,6 +283,10 @@ class OpenAIProvider(Provider):
|
||||
)
|
||||
return self._dimension
|
||||
|
||||
# Transient retry intentionally covers generation too (RAG sampling path):
|
||||
# a pod rollover breaks generation as readily as embedding. Worst case adds
|
||||
# ~30s (5 attempts, 2s→60s backoff) to an interactive call hitting a
|
||||
# sustained connection issue, which is preferable to a hard failure.
|
||||
@_retry_transient
|
||||
async def generate(self, prompt: str, max_tokens: int = 500) -> str:
|
||||
"""
|
||||
|
||||
@@ -64,14 +64,18 @@ def _drop_reason(exc: BaseException) -> str:
|
||||
Distinguishes the transient backend-pod-rollover causes (connection /
|
||||
timeout — the ones provider-level retry should now ride through, card 309)
|
||||
from persistent faults, so ``astrolabe_vector_ingest_dropped_total`` is
|
||||
alertable per cause. Unwraps a single ExceptionGroup leaf. Best-effort:
|
||||
alertable per cause. Descends through nested ExceptionGroups to the first
|
||||
leaf so a doubly-wrapped cause isn't mislabelled ``other``. Best-effort:
|
||||
unknown causes fall back to ``other``.
|
||||
"""
|
||||
# An anyio task group can wrap the real cause; classify the first leaf.
|
||||
if isinstance(exc, BaseExceptionGroup) and exc.exceptions:
|
||||
# An anyio task group can wrap the real cause (and nest groups when sub-tasks
|
||||
# use their own groups); descend to the first concrete leaf.
|
||||
while isinstance(exc, BaseExceptionGroup) and exc.exceptions:
|
||||
exc = exc.exceptions[0]
|
||||
|
||||
# httpx transport errors (raised by the OpenAI/gateway client underneath).
|
||||
# Raw httpx transport errors from direct Nextcloud API calls (the nc_client
|
||||
# uses httpx directly); the openai checks below catch the SDK-wrapped
|
||||
# variants of the same failure modes.
|
||||
if isinstance(exc, httpx.TimeoutException):
|
||||
return "timeout"
|
||||
if isinstance(exc, httpx.ConnectError):
|
||||
@@ -387,6 +391,13 @@ async def process_document(
|
||||
(3) suits the in-process SQLite pool, which has no durable retry. The
|
||||
procrastinate worker passes ``1`` so durable retry is owned by the
|
||||
queue (and survives worker crashes), avoiding compounding 3×N retries.
|
||||
|
||||
Retry layering: the embedding provider adds its own transient retry (5
|
||||
attempts, 2s→60s backoff — card 309) *inside* each of these attempts. On the
|
||||
in-process path (max_retries=3) a sustained outage therefore costs up to
|
||||
5×3=15 provider calls (~90s wall-clock) before the document is dropped and
|
||||
re-picked on the next scan; the procrastinate path (max_retries=1) caps it at
|
||||
one outer attempt (~30s) and defers. Don't stack a third retry layer here.
|
||||
"""
|
||||
start_time = time.time()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user