feat(embedding): gateway provider discovers dimension via GET /v1/models
External-mode tenant pods CrashLoop at startup: Qdrant collection init calls get_dimension() before any embed(), but GatewayProvider only learns its dimension lazily after the first embed, and the gateway model isn't an OpenAI model so the base class can't know it statically. - Add GatewayProvider._detect_dimension() — the async startup hook the vector-sync bootstrap already invokes (vector/qdrant_client.py: hasattr(provider, "_detect_dimension")) for Ollama. It GETs the gateway's GET /v1/models and sets _dimension from the entry whose id matches the configured model. Best-effort: any failure (old gateway, model absent, network) leaves _dimension unset so the inherited lazy detect-on-first-embed still applies — never fatal. Presents the M2M bearer when configured. - Switch the default embedding_gateway_model to the gateway's provider- namespaced id "mistral/mistral-embed" (the gateway routes on the "/"-prefix and sends "mistral-embed" upstream); collapse a duplicated config field. Pairs with astrolabe-cloud-website#229 (gateway /v1/models, namespaced ids). Tests: discovery sets dim w/o embed, sends bearer, non-fatal on 404/absent/error, skips when already known.
This commit is contained in:
@@ -147,6 +147,66 @@ class GatewayProvider(OpenAIProvider):
|
||||
if self._token_provider is not None:
|
||||
self.client.api_key = await self._token_provider.get_token()
|
||||
|
||||
async def _detect_dimension(self) -> None:
|
||||
"""Resolve the embedding dimension from the gateway's ``GET /v1/models``
|
||||
before the first embed.
|
||||
|
||||
Qdrant collection init needs the vector size at startup (it calls
|
||||
``get_dimension()`` before any ``embed()``); the vector-sync bootstrap
|
||||
invokes this hook first (``vector/qdrant_client.py`` —
|
||||
``hasattr(provider, "_detect_dimension")``). The gateway is the
|
||||
authority on the dimensions of the models it serves, so we read it from
|
||||
there rather than hardcoding (``mistral-embed`` isn't an OpenAI model,
|
||||
so the OpenAI-wire base class can't know its size statically).
|
||||
|
||||
Best-effort: any failure (old gateway without /v1/models, model absent,
|
||||
network) leaves ``_dimension`` unset so the inherited lazy
|
||||
detect-on-first-embed path still applies. Never raises.
|
||||
"""
|
||||
if self._dimension is not None:
|
||||
return # already known (e.g. an OpenAI model in the static map)
|
||||
|
||||
# ``models`` is a sibling of ``embeddings`` under the gateway's base —
|
||||
# derive it from the same base_url the OpenAI client uses for embeds so
|
||||
# the two stay consistent (str(base_url) has a trailing slash).
|
||||
models_url = str(self.client.base_url).rstrip("/") + "/models"
|
||||
headers: dict[str, str] = {}
|
||||
if self._token_provider is not None:
|
||||
headers["Authorization"] = (
|
||||
f"Bearer {await self._token_provider.get_token()}"
|
||||
)
|
||||
try:
|
||||
async with httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(10.0, connect=5.0)
|
||||
) as client:
|
||||
resp = await client.get(models_url, headers=headers)
|
||||
resp.raise_for_status()
|
||||
catalog = resp.json().get("data", [])
|
||||
for entry in catalog:
|
||||
if entry.get("id") == self.embedding_model:
|
||||
dim = entry.get("dimension")
|
||||
if isinstance(dim, int):
|
||||
self._dimension = dim
|
||||
logger.info(
|
||||
"Resolved embedding dimension %d for model %s via "
|
||||
"gateway /v1/models",
|
||||
dim,
|
||||
self.embedding_model,
|
||||
)
|
||||
return
|
||||
logger.warning(
|
||||
"Gateway /v1/models reported no dimension for model %s; "
|
||||
"falling back to lazy detection on first embed",
|
||||
self.embedding_model,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - best-effort, never fatal
|
||||
logger.warning(
|
||||
"Could not fetch model dimensions from gateway %s: %s; "
|
||||
"falling back to lazy detection on first embed",
|
||||
models_url,
|
||||
exc,
|
||||
)
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
await self._ensure_bearer()
|
||||
return await super().embed(text)
|
||||
|
||||
Reference in New Issue
Block a user