fix(vector): retry transient embed errors so a pod rollover drops 0 docs

From card 309 (OHR-Bench smoke-test triage): during a backend-pod rollover the
embedding endpoint was briefly unreachable, and openai.APIConnectionError /
ConnectError propagated unretried (the provider only retried 429). Documents
exhausted the 3 in-process retries and were dropped for that scan cycle.

Broaden the provider-level retry to the transient set -- APIConnectionError,
APITimeoutError, 429, and 5xx -- on the existing exponential backoff (2s->60s,
5 attempts), so a few seconds of retry rides through the rollover. Permanent
4xx (auth, bad request) still re-raise immediately. Generalize the shared
_retry helper (retry_on_rate_limit -> retry_on_transient, predicate renamed to
should_retry, accurate log label) with a back-compat alias; Mistral gets 429+5xx
for parity. The production gateway path inherits this via GatewayProvider, which
delegates to the decorated OpenAIProvider methods.

Add astrolabe_vector_ingest_dropped_total{reason}, incremented when a document
exhausts retries, classified (connection|timeout|rate_limit|server|qdrant|other)
by _drop_reason so the embed-drop rate is alertable per cause. Dropped docs are
NOT marked failed, so the next full scan re-picks them (re-queue via scan loop).

Refs: Deck board 12 card 309 (AC #1 no permanently-dropped docs; embed-drop
metric for AC #5).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

nextcloud_mcp_server/providers/openai.py

@@ -8,16 +8,37 @@ Supports:
 
 import logging
 
-from openai import AsyncOpenAI, RateLimitError
+from openai import APIConnectionError, APIError, APIStatusError, AsyncOpenAI
 
-from ._retry import retry_on_rate_limit
+from ._retry import retry_on_transient
 from .base import Provider
 
 logger = logging.getLogger(__name__)
 
-# OpenAI's RateLimitError is itself a 429-specific class, so the default
-# is_rate_limit predicate ("always True") matches the previous behavior.
-_retry_429 = retry_on_rate_limit(RateLimitError, provider_name="OpenAI")
+
+def _is_transient(exc: BaseException) -> bool:
+    """Whether an OpenAI APIError is transient and worth retrying.
+
+    Covers the failures seen dropping documents during a backend-pod rollover
+    (card 309): ``APIConnectionError`` / ``APITimeoutError`` (brief gateway
+    unreachability) and 429 / 5xx status errors. Permanent 4xx (auth, bad
+    request) are NOT retried — they would fail identically every attempt.
+    """
+    if isinstance(exc, APIConnectionError):  # incl. APITimeoutError
+        return True
+    if isinstance(exc, APIStatusError):
+        return exc.status_code == 429 or exc.status_code >= 500
+    return False
+
+
+# Catch the APIError base (parent of both APIConnectionError and APIStatusError)
+# and let the predicate decide; non-transient errors re-raise immediately.
+_retry_transient = retry_on_transient(
+    APIError,
+    should_retry=_is_transient,
+    provider_name="OpenAI",
+    label="transient error",
+)
 
 
 # Well-known embedding dimensions for OpenAI models
@@ -95,7 +116,7 @@ class OpenAIProvider(Provider):
         """Whether this provider supports text generation."""
         return self.generation_model is not None
 
-    @_retry_429
+    @_retry_transient
     async def embed(self, text: str) -> list[float]:
         """
         Generate embedding vector for text.
@@ -208,7 +229,7 @@ class OpenAIProvider(Provider):
 
         return all_embeddings, total_tokens
 
-    @_retry_429
+    @_retry_transient
     async def _embed_batch_request(
         self, batch: list[str]
     ) -> tuple[list[list[float]], int]:
@@ -262,7 +283,7 @@ class OpenAIProvider(Provider):
             )
         return self._dimension
 
-    @_retry_429
+    @_retry_transient
     async def generate(self, prompt: str, max_tokens: int = 500) -> str:
         """
         Generate text from a prompt.