fix(vector): retry transient embed errors so a pod rollover drops 0 docs

From card 309 (OHR-Bench smoke-test triage): during a backend-pod rollover the
embedding endpoint was briefly unreachable, and openai.APIConnectionError /
ConnectError propagated unretried (the provider only retried 429). Documents
exhausted the 3 in-process retries and were dropped for that scan cycle.

Broaden the provider-level retry to the transient set -- APIConnectionError,
APITimeoutError, 429, and 5xx -- on the existing exponential backoff (2s->60s,
5 attempts), so a few seconds of retry rides through the rollover. Permanent
4xx (auth, bad request) still re-raise immediately. Generalize the shared
_retry helper (retry_on_rate_limit -> retry_on_transient, predicate renamed to
should_retry, accurate log label) with a back-compat alias; Mistral gets 429+5xx
for parity. The production gateway path inherits this via GatewayProvider, which
delegates to the decorated OpenAIProvider methods.

Add astrolabe_vector_ingest_dropped_total{reason}, incremented when a document
exhausts retries, classified (connection|timeout|rate_limit|server|qdrant|other)
by _drop_reason so the embed-drop rate is alertable per cause. Dropped docs are
NOT marked failed, so the next full scan re-picks them (re-queue via scan loop).

Refs: Deck board 12 card 309 (AC #1 no permanently-dropped docs; embed-drop
metric for AC #5).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nextcloud_mcp_server/providers/_retry.py

@@ -1,8 +1,10 @@
-"""Shared rate-limit retry helper for provider modules.
+"""Shared transient-error retry helper for provider modules.
 
-OpenAI and Mistral both retry on 429 with the same exponential-backoff curve;
-extracting the loop here keeps the two provider modules thin and lets future
-providers (Bedrock throttling, etc.) reuse the same primitive.
+OpenAI and Mistral retry transient failures (429 rate limits, plus connection
+drops / timeouts / 5xx for the embedding path) on the same exponential-backoff
+curve; extracting the loop here keeps the two provider modules thin and lets
+future providers (Bedrock throttling, etc.) reuse the same primitive. The
+``should_retry`` predicate decides which caught exceptions are transient.
 """
 
 from __future__ import annotations
@@ -23,23 +25,26 @@ MAX_RETRY_DELAY = 60.0
 T = TypeVar("T")
 
 
-def retry_on_rate_limit(
-    exception_type: type[BaseException],
-    is_rate_limit: Callable[[BaseException], bool] = lambda _exc: True,
+def retry_on_transient(
+    exception_type: type[BaseException] | tuple[type[BaseException], ...],
+    should_retry: Callable[[BaseException], bool] = lambda _exc: True,
     *,
     provider_name: str = "provider",
+    label: str = "rate limit",
 ) -> Callable[[Callable[..., Awaitable[T]]], Callable[..., Awaitable[T]]]:
-    """Build a decorator that retries on rate-limit exceptions.
+    """Build a decorator that retries transient exceptions with backoff.
 
     Args:
-        exception_type: Catch this exception class (e.g. ``openai.RateLimitError``,
-            ``mistralai.client.errors.SDKError``).
-        is_rate_limit: Predicate that decides whether a caught exception is
-            actually a rate-limit (vs. some other error of the same class).
-            Defaults to "always True" — appropriate when ``exception_type`` is
-            already a rate-limit-specific class.
+        exception_type: Catch this exception class (or tuple of classes), e.g.
+            ``openai.APIError`` or ``mistralai.client.errors.SDKError``.
+        should_retry: Predicate that decides whether a caught exception is
+            transient (and so retryable) vs. a permanent error of the same
+            class. Defaults to "always True" — appropriate when
+            ``exception_type`` is already transient-specific (e.g. a 429 class).
         provider_name: Used in log messages so operators can tell which
             provider exhausted retries.
+        label: Short noun for the log message ("rate limit", "transient error")
+            so the line accurately names what was retried.
     """
 
     def decorator(func: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]:
@@ -52,22 +57,27 @@ def retry_on_rate_limit(
                 try:
                     return await func(*args, **kwargs)
                 except exception_type as e:
-                    if not is_rate_limit(e):
+                    if not should_retry(e):
                         raise
                     last_error = e
                     if attempt < MAX_RETRIES:
                         logger.warning(
-                            "%s rate limit hit (attempt %d/%d), retrying in %.1fs...",
+                            "%s %s (attempt %d/%d): %r; retrying in %.1fs...",
                             provider_name,
+                            label,
                             attempt,
                             MAX_RETRIES,
+                            e,
                             retry_delay,
                         )
                         await anyio.sleep(retry_delay)
                         retry_delay = min(retry_delay * 2, MAX_RETRY_DELAY)
 
             logger.error(
-                "%s rate limit exceeded after %d attempts", provider_name, MAX_RETRIES
+                "%s %s not resolved after %d attempts",
+                provider_name,
+                label,
+                MAX_RETRIES,
             )
             if last_error is None:  # pragma: no cover — loop above always sets this
                 raise RuntimeError("retry loop exited without capturing an error")
@@ -76,3 +86,7 @@ def retry_on_rate_limit(
         return wrapper
 
     return decorator
+
+
+# Back-compat alias: the helper was originally rate-limit-specific.
+retry_on_rate_limit = retry_on_transient