Chris CoutinhoandClaude Opus 4.8
258ee96f4c
From card 309 (OHR-Bench smoke-test triage): during a backend-pod rollover the
embedding endpoint was briefly unreachable, and openai.APIConnectionError /
ConnectError propagated unretried (the provider only retried 429). Documents
exhausted the 3 in-process retries and were dropped for that scan cycle.
Broaden the provider-level retry to the transient set -- APIConnectionError,
APITimeoutError, 429, and 5xx -- on the existing exponential backoff (2s->60s,
5 attempts), so a few seconds of retry rides through the rollover. Permanent
4xx (auth, bad request) still re-raise immediately. Generalize the shared
_retry helper (retry_on_rate_limit -> retry_on_transient, predicate renamed to
should_retry, accurate log label) with a back-compat alias; Mistral gets 429+5xx
for parity. The production gateway path inherits this via GatewayProvider, which
delegates to the decorated OpenAIProvider methods.
Add astrolabe_vector_ingest_dropped_total{reason}, incremented when a document
exhausts retries, classified (connection|timeout|rate_limit|server|qdrant|other)
by _drop_reason so the embed-drop rate is alertable per cause. Dropped docs are
NOT marked failed, so the next full scan re-picks them (re-queue via scan loop).
Refs: Deck board 12 card 309 (AC #1 no permanently-dropped docs; embed-drop
metric for AC #5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
73 lines
2.1 KiB
Python
73 lines
2.1 KiB
Python
"""Unit tests for the embed-drop classifier (card 309).
|
|
|
|
``processor._drop_reason`` maps a terminal indexing failure to a metric label
|
|
so the transient backend-pod-rollover causes (connection / timeout) are
|
|
alertable on ``astrolabe_vector_ingest_dropped_total`` distinctly from
|
|
persistent faults.
|
|
"""
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
from nextcloud_mcp_server.vector import processor
|
|
|
|
|
|
def _req() -> httpx.Request:
|
|
return httpx.Request("POST", "http://gw/v1/embeddings")
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_httpx_connect_and_timeout_classified():
|
|
assert processor._drop_reason(httpx.ConnectError("refused")) == "connection"
|
|
assert processor._drop_reason(httpx.ReadTimeout("slow")) == "timeout"
|
|
assert processor._drop_reason(httpx.ConnectTimeout("slow")) == "timeout"
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_openai_errors_classified():
|
|
from openai import (
|
|
APIConnectionError,
|
|
APITimeoutError,
|
|
InternalServerError,
|
|
RateLimitError,
|
|
)
|
|
|
|
req = _req()
|
|
assert processor._drop_reason(APIConnectionError(request=req)) == "connection"
|
|
assert processor._drop_reason(APITimeoutError(request=req)) == "timeout"
|
|
assert (
|
|
processor._drop_reason(
|
|
RateLimitError("rl", response=httpx.Response(429, request=req), body=None)
|
|
)
|
|
== "rate_limit"
|
|
)
|
|
assert (
|
|
processor._drop_reason(
|
|
InternalServerError(
|
|
"boom", response=httpx.Response(503, request=req), body=None
|
|
)
|
|
)
|
|
== "server"
|
|
)
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_exception_group_unwraps_to_leaf():
|
|
group = BaseExceptionGroup(
|
|
"unhandled errors in a TaskGroup", [httpx.ConnectError("refused")]
|
|
)
|
|
assert processor._drop_reason(group) == "connection"
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_qdrant_namespace_classified():
|
|
from qdrant_client.http.exceptions import UnexpectedResponse
|
|
|
|
exc = UnexpectedResponse(500, "err", b"", headers=None)
|
|
assert processor._drop_reason(exc) == "qdrant"
|
|
|
|
|
|
@pytest.mark.unit
|
|
def test_unknown_error_falls_back_to_other():
|
|
assert processor._drop_reason(ValueError("nope")) == "other"
|